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Preface 


Since  1998,  the  Army’s  Training  and  Doctrine  Command  (TRADOC) 
has  been  engaged  in  establishing  and  fielding  The  Army  Distributed 
Learning  Program  (TADLP).  The  intent  of  this  program  is  to  foster 
and  provide  for  the  use  of  digital  technologies  to  enhance  and  extend 
traditional  methods  of  learning  within  the  Army’s  institutional  train¬ 
ing  strategy.  Distributed  learning  (DL)  is  intended  to  speed  the  pace  of 
learning  and  allow  training  to  take  place  when  and  where  soldiers  need 
it,  and  the  Army  has  an  expansive  vision  for  a  greatly  increased  role  for 
DL  over  time.  For  FY  2007,  TRADOC  asked  RAND  Arroyo  Center 
to  assess  how  efficiently  and  effectively  TADLP  had  accomplished  its 
objectives,  and  to  provide  recommendations  for  improvements  to  the 
program.  This  project  continued  in  FY  2008  with  a  focus  on  strategic 
improvements  to  TADLP  that  would  allow  the  Army  to  leverage  DL 
more  effectively  in  the  future. 

This  document  reports  on  Arroyo’s  assessment  of  the  progress 
TADLP  has  made  toward  its  goals,  and  it  develops  options  for  improv¬ 
ing  the  program.  The  strategic  examination  focused  on  DL’s  role  in 
supporting  the  execution  of  the  Army  Leader  Development  Program 
and  the  transformation  of  the  Noncommissioned  Officer  Education 
System  (NCOES). 

This  report  will  be  of  interest  to  those  involved  in  planning,  devel¬ 
oping,  delivering,  and  evaluating  interactive  multimedia  instruction 
(IMI)  and  other  forms  of  distributed  learning.  It  will  also  be  of  interest 
to  those  involved  in  education  and  training  more  generally,  strategic 
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planning  for  education  and  training,  and  training  transformation  and 
training  system  integration  within  the  Department  of  Defense. 

The  Project  Unique  Identification  Codes  (PUICs)  for  the  research 
that  produced  this  document  are  ATFCR08597  and  ATFCR07213. 

This  research  was  sponsored  by  TRADOC  and  conducted  within 
RAND  Arroyo  Center’s  Manpower  and  Training  Program.  RAND 
Arroyo  Center,  part  of  the  RAND  Corporation,  is  a  federally  funded 
research  and  development  center  sponsored  by  the  United  States  Army. 


For  more  information  on  RAND  Arroyo  Center,  contact  the 
Director  of  Operations  (telephone  310-393-0411,  extension  6419;  fax 
310-451-6952;  email  Marcy_Agmon@rand.org),  or  visit  Arroyo’s  web¬ 
site  at  http://www.rand.org/ard.html. 
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Summary 


The  Army  Distributed  Learning  Program  (TADLP),  established  in 
1998,  is  part  of  the  Army’s  training  and  leader  development  system 
and  supports  execution  of  the  Army’s  training  and  leader  development 
strategies.  Distributed  learning  (DL)  enhances  and  extends  traditional 
approaches  to  learning  by  making  use  of  multiple  means  and  technolo¬ 
gies  to  enable  the  delivery  of  training  and  learning  wherever  and  when¬ 
ever  soldiers  and  leaders  need  it. 

DL  capabilities,  especially  the  ability  to  provide  learning  “any¬ 
time,  anyplace,”  are  becoming  increasingly  important  in  supporting 
the  Army’s  training  and  leader  development  system.  Requirements  are 
expanding,  as  is  the  need  to  export  training  and  learning  to  satisfy 
those  requirements.  Training  and  education  programs  that  support 
enhanced  leader  adaptability  and  complex  thinking  skills  are  espe¬ 
cially  needed.  The  training  system  also  has  to  address  such  factors  as 
the  demands  of  equipment  modernization,  modularity,  increases  in  the 
size  of  the  Army,  and  the  current  operating  environment.  Thus,  both 
the  amount  and  the  complexity  of  needed  training  have  increased. 
Moreover,  the  demands  of  supporting  major  operational  requirements, 
including  Afghanistan  and  Iraq,  have  limited  the  Army’s  ability  to 
increase  the  time  soldiers  spend  in  institutional  schoolhouses. 

The  need  to  expand  the  amount  of  exportable  learning  has  been 
further  emphasized  by  Army  Force  Generation  (ARFORGEN),  a 
cyclical  readiness  process  that  places  great  demand  on  the  scheduling 
of  institutional  courses  and  limits  the  length  of  time  that  soldiers  can 
spend  at  a  schoolhouse  (away  from  home  station).  To  support  ARFOR- 
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GEN,  much  of  institutional  training  must  be  completed  in  the  narrow 
window  (of  about  six  months  for  the  active  component  [AC]  and  one 
year  for  the  reserve  component  [RC])  between  the  time  a  unit  returns 
from  an  operational  deployment  and  when  it  begins  collective  train¬ 
ing.  Given  this  context,  DL  has  been  recognized  as  an  increasingly 
important  part  of  the  Army’s  training  and  leader  development  strategy, 
and  the  Army  has  identified  the  need  to  transform  training  and  leader 
development  programs  in  a  major  way  through  increased  use  of  DL. 

Despite  a  growing  recognition  of  the  role  to  be  played  by  DL, 
resources  for  producing  courseware  within  TADLP  are  limited  and 
declining.  In  2008,  TADLP  received  only  enough  funding  to  develop 
a  small  fraction  of  the  total  institutional  requirement,  further,  as  of 
January  2008,  funding  for  TADLP  for  LY  2006  through  LY  2011 
had  decreased  by  40  percent  from  the  amount  budgeted  three  years 
earlier.  Moreover,  budget  figures  reveal  that  the  production  of  DL 
courseware  is  receiving  less  emphasis  over  time,  with  a  greater  pro¬ 
portion  of  TADLP  funds  going  toward  the  Army  Learning  Manage¬ 
ment  System  (ALMS)  and  DL  classrooms.  Considering  the  growing 
importance  of  DL  within  the  Army’s  training,  in  LY  2007-2008,  the 
Army’s  Training  and  Doctrine  Command  (TRADOC)  asked  RAND 
Arroyo  Center  to  assess  TADLP’s  performance  and  provide  options  for 
improvement,  both  in  the  near  and  longer  terms.  By  agreement  with 
the  sponsor,  the  project  focused  on  the  courseware  that  most  directly 
supports  readiness. 


Methodology  and  Approach 

The  study  was  done  in  three  stages,  the  first  two  focusing  on  the  near 
term,  and  the  third  focusing  on  the  longer  term.  In  the  first  stage  (see 
Chapter  Three),  we  used  data  from  f  Y  2006  (and  informal  spot  checks 
in  f  Y  2007-2008  to  ensure  continued  validity)  to  assess  TADLP’s  pro¬ 
gram  for  developing  interactive  multimedia  instruction  (IMI)  course¬ 
ware  against  five  measures  of  effectiveness  for  readiness-related  courses: 
impact,  efficiency,  quality,  cycle  time,  and  responsiveness.  In  the  second 
stage  (see  Chapter  Pour),  we  developed  options  for  improving  the  IMI 
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program  (as  it  existed  up  through  2008)  to  address  the  areas  of  weak¬ 
ness  identified  in  the  first  stage.  Finally,  in  the  third  stage  (see  Chapters 
Five  through  Seven),  we  developed  and  outlined  options  for  broaden¬ 
ing  the  current  TADLP  beyond  its  mainly  IMI  focus  to  increase  its 
impact,  quality,  and  responsiveness,  and  for  improving  efficiency. 

The  study  drew  upon  a  variety  of  methods  and  sources,  including 
reviews  of  relevant  policy  and  program  documents;  analysis  of  Army 
institutional  course  management  data  and  other  databases;  project- 
developed  surveys  concerning  specific  DL  courses;  interviews  and  focus 
groups  with  proponent  schools,  DL  contractors,  and  TRADOC  head¬ 
quarters  staff;  reviews  of  Army  processes  for  developing  courseware; 
and  an  analysis  of  the  quality  of  selected  IMI  courseware. 


Conclusions 

TADLP  Courseware  Has  Had  a  Narrow  Focus  That  Limits  Its  Potential 

The  study’s  examination  of  the  state  of  TADLP  found  that  up  through 
2008  the  program  has  had  a  relatively  narrow  focus  on  one  approach  to 
DL.  With  a  small  budget  and  limited  support  in  TRADOC  for  moving 
away  from  traditional  residential  approaches,  the  program  has  focused 
primarily  on  producing  stand-alone  IMI  (i.e.,  stand-alone  computer- 
based  instruction  not  needing  instructor  support),  which  focuses  on 
the  learning  and  comprehension  of  facts,  concepts,  and  procedures  in 
preparation  for  resident  training. 

Our  Assessment  Showed  a  Need  for  TADLP  Improvement  with 
Regard  to  All  Measures  of  Effectiveness  (MOEs) 

•  Limited  impact  on  training.  After  nearly  a  decade  of  operation, 
DL  training  represents  only  a  small  fraction  of  the  total  amount 
of  institutional  training.  In  FY  2006,  only  around  100  of  the  22 7 
funded  Army  DL  modules  in  the  highest-priority  categories  (i.e., 
reclassification  courses,  Noncommissioned  Officer  Education 
System  (NCOES)  courses,  RC  Captains  Career  Courses,  addi¬ 
tional  skill  identifier/special  qualifications  identifier  [ASI/SQI] 
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courses)  had  any  enrollees,  and  the  majority  of  those  that  were  in 
use  produced  less  than  200  graduates  in  FY  2006.  Moreover,  out¬ 
side  of  ASI/SQI  courses,  DL  accounted  for  less  than  6  percent  of 
all  instruction  in  high-priority  courses,  with  a  considerably  lower 
percentage  for  AC  soldiers. 

•  Signs  of  inefficiency.  The  efficiency  of  the  Army’s  program  has 
been  limited  by  a  large  percentage  of  funded  courses  that  do  not 
achieve  enrollments  or  are  used  for  only  a  few  years  after  ini¬ 
tial  development  is  complete.  The  program  is  also  limited  by  a 
relatively  small  number  of  enrollees  per  class,  and  low  graduation 
rates. 

•  Concerns  about  courseware  quality.  We  found  no  efforts  in 
TADLP  through  FY  2008  to  assess  course  quality  at  the  program 
level.  Therefore,  Arroyo  developed  an  approach  to  IMI  evalua¬ 
tion  that  focused  on  technical  criteria  and  instructional  design  of 
content.  The  application  of  this  methodology  to  a  small  sample 
of  DL  courses  revealed  some  potentially  significant  issues  about 
the  quality  of  TADLP  courseware,  especially  with  regard  to  the 
pedagogical  characteristics  of  the  instruction. 

•  Long  cycle  times.  Through  2008,  the  amount  of  time  required 
to  produce  a  DL  course  under  TADLP  has  been  overly  long.  The 
average  course  production  time,  from  the  need  identification  to 
the  first  student  use,  is  nearly  3.5  years.  In  some  cases,  DL  courses 
have  been  declared  obsolete  even  before  they  could  be  completed. 
In  the  commercial  world,  a  comparable  cycle  time  is  estimated  at 
less  than  a  year. 

•  Lack  of  responsiveness.  Responsiveness  is  the  ease  with  which 
courseware  can  be  adapted  in  response  to  changing  requirements. 
During  the  period  of  the  study,  the  Army  used  the  simplest,  least 
flexible  approach  in  its  acquisition  strategy  for  acquiring  IMI 
courseware.  That  acquisition  strategy,  coupled  with  a  policy  that 
made  it  difficult  to  update  courseware  on  a  timely  basis,  made  DL 
courseware  unresponsive  to  changing  requirements. 
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TADLP  Lacks  a  Structured  Process  for  Evaluation,  Assessment  and 
Improvement 

A  major  issue  with  TADLP  is  the  lack  of  an  overall  process  and  support¬ 
ing  data  for  evaluation.  This  shortcoming  severely  impedes  the  Army’s 
ability  to  identify  the  underlying  causes  of  the  shortfalls  in  TADLP 
performance,  design  effective  improvement  initiatives,  and  verify  their 
success.  Moreover,  our  assessment  shows  that  such  a  process  is  feasible. 

The  Potential  Exists  to  Significantly  Expand  the  Role  of  DL  in  Army 
Training 

Our  research  concluded  that  there  is  great  potential  for  DL  expansion, 
provided  that  TADLP  is  able  to  move  beyond  the  almost  exclusive 
focus  on  stand-alone  IMI  to  deliver  instruction.  The  program  can  use 
DL  both  to  provide  more  training  and  to  train  more-complex  skills. 
Current  practices  in  industry,  in  academia,  and  even  in  some  emerg¬ 
ing  Army  initiatives  suggest  that  the  Army  can  make  a  much  greater 
proportion  of  structured  training  flexible  and  exportable,  especially  for 
the  AC. 


Recommendations  for  Improving  the  IMI  Program 

In  the  course  of  our  examination  we  gained  an  understanding  of  some 
of  the  key  factors  that  underlie  the  areas  needing  improvement.  We 
identified  five  near-term  initiatives  that  would  increase  the  impact 
of  the  Army’s  IMI  program,  increase  the  quality  of  the  product,  and 
improve  the  efficiency  and  responsiveness  of  the  process. 

Add  flexibility  to  the  courseware  acquisition  strategy.  The 
Army’s  acquisition  strategy  could  be  made  more  flexible  and  effective 
by  focusing  on  requirements  contracts — which  would  “prequalify”  a  set 
of  prime  contractors  for  DL  development  in  focused  areas — and  by 
selectively  using  both  a  systems  (as  opposed  to  product)  output  phi¬ 
losophy  during  acquisition  planning  and  an  incremental  acquisition 
approach  for  complex  IMI.  Decentralizing  contract  management  and 
administration  could  lead  to  further  increases  in  responsiveness. 
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Ensure  sufficient  resources  per  training  module  for  stand¬ 
alone  IMI.  The  Army’s  approach  to  developing  IMI  has  assumed  that 
schools  can  adequately  support  courseware  development,  which  was 
seldom  the  case.  When  such  support  was  not  available,  the  result  was 
an  extended  cycle  time  and  compromises  in  courseware  quality.  We 
recommend  that  the  Army  add  more  resources  when  needed  on  a  per- 
course  basis  to  cover  such  categories  as  subject  matter  expert  support 
for  development  and  instructor  support  to  students  during  course 
delivery.  In  the  short  run,  this  would  mean  funding  fewer  DL  courses, 
but  the  improved  outcome  will  eventually  lead  to  an  increased  return 
on  investment  through  greater  usage  of  DL  content. 

Undertake  systematic  process  improvements  to  reduce  IMI 
cycle  times  for  production.  There  are  a  number  of  known  issues  that 
appear  to  have  contributed  to  long  development  times,  including  too 
many  steps  and  required  signoffs  in  the  production  process.  Although 
TRADOC  has  implemented  a  number  of  initiatives  to  reduce  cycle 
time,  additional  improvements  are  possible  through  such  actions  as 
releasing  IMI  funds  at  the  beginning  of  the  year  and  continuing  and 
refining  the  application  of  process  improvement  methods. 

Increase  local  participation  in  IMI  production  and  contract 
administration.  To  achieve  multiple  improvements  in  program  out¬ 
comes,  we  recommend  two  ways  for  TADLP  to  increase  proponent 
participation  in  IMI  production.  First,  TADLP  should  increase  the 
practice  of  producing  some  IMI  in  house.  Proper  selection  of  content 
for  local  production  of  IMI  can  reduce  variable  costs  and  cycle  time, 
and  it  can  also  increase  IMI's  responsiveness  to  the  need  for  change. 
Second,  in  cases  where  IMI  production  must  still  be  contracted  out, 
TADLP  should  decentralize  selected  aspects  of  contract  management 
and  administration  in  order  to  increase  the  responsiveness  and  the 
quality  of  the  IMI  product. 

Institute  a  program-level  IMI  quality  evaluation  component 
to  support  TADLP  improvements.  A  courseware  quality  assessment 
focusing  on  the  instructional  design  and  technical  features  of  IMI 
courseware  could  be  accomplished  by  TADLP  with  relatively  modest 
resources.  The  results  could  form  the  basis  for  program-level  improve¬ 
ments  in  DL.  Other  types  of  course  quality  evaluations  should  also  be 
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added  over  time,  such  as  assessing  courseware  currency,  learner  reac¬ 
tions,  and  knowledge  retention. 


Recommendations  to  Broaden  TADLP  to  Better  Support 
Training  and  Leader  Development  Requirements 

An  overall  finding  from  our  analysis  is  that  TADLP  plays  an  unneces¬ 
sarily  limited  role  in  support  of  the  Army’s  programs  for  training  and 
leader  development.  Leveraging  expanded  DL  capabilities  (and  mobile 
training  team  (MTT))  methods  has  the  potential  to  create  a  much 
more  exportable  training  capacity  in  support  of  leader  development 
and  the  Reset  cycle  under  ARFORGEN.  Such  changes  could  help  to 
significantly  reduce  the  length  of  residential  courses,  better  support 
unit  readiness,  and  reduce  the  backlog  for  NCOES  courses.  Below  we 
outline  three  recommendations  for  broadening  TADLP  to  accomplish 
more. 

Employ  Blended  Learning  Options  to  Significantly  Expand  DL's  Role 

We  recommend  that  the  Army  use  a  broader  range  of  DL  options  in 
addition  to  stand-alone  IMI.  A  mix  of  approaches,  such  as  those  shown 
in  Table  S.l,  can  be  used  to  extend  the  reach  of  DL.  For  example, 
key  mechanisms  supporting  a  more  widespread  transformation  include 
evolution  of  an  IMI  approach  that  allows  greater  instructor  support 
and  greater  use  of  collaborative  DL  to  replace  a  large  portion  of  resident 
classroom  discussion  hours. 

Many  private-sector  organizations  requiring  DL  are  using  a  mix 
of  approaches  within  a  single  course — resulting  in  a  “blended  learning” 
approach,  which  not  only  combines  DL  with  face-to-face  instruction, 
but  also  blends  various  types  of  DL.  This  new  type  of  course  is  con¬ 
structed  by  considering  all  potential  modalities  for  delivery  of  training 
on  a  task-by-task  basis,  leading  to  a  distribution  of  different  modalities 
across  an  entire  course. 

A  few  courses  in  the  Army  (although  outside  of  TADLP)  are  also 
beginning  to  use  blended  learning  methods.  For  example,  the  applica¬ 
tion  of  blended  learning  options  reduced  the  7.5-week  Special  Forces 
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Table  S.1 

A  Wider  Range  of  Options  for  DL 


Option 

Description 

IMI  Levels  1-3,  Stand-Alone 

Allows  students  to  access  courses  anywhere, 
anytime;  instruction  is  embedded  in  the 
technology,  thus  requiring  limited  interaction  or 
collaboration  with  instructors 

IMI  Levels  1-3,  Instructor 
Supported 

Uses  technology  to  deliver  content,  but  makes 
instructors  available  to  monitor  and  support 
learner  progress,  respond  to  student  questions, 
and  provide  feedback 

IMI  Level  4,  Asynchronous 

Uses  "serious  games"  to  provide  learners  with 
an  immersive  experience  at  high  levels  of 
interactivity 

IMI  Level  4,  Synchronous 

Involves  simulations  and  serious  games  using 
multi-sided  interactions  with  two  or  more  players 
or  multiple  role  players,  as  well  as  artificial 
intelligence  and  instructors 

Asynchronous  Collaborative  DL 

Uses  technology  to  enable  communication 
between  student  and  instructor  and  among 
students  via  email,  forums,  discussion  boards, 
telephones,  wikis,  etc. 

Synchronous  Collaborative  DL 

Involves  real-time  collaboration  between 
students  and  instructors 

Mobile  Training  Teams  (MTTs) 

Provides  face-to-face  instruction  to  learners  at 
their  locations  rather  than  at  the  schoolhouse; 
though  not  technically  a  DL  approach,  this 
method  can  be  used  to  export  training 

NOTE:  In  general,  IMI  levels  refer  to  the  degree  the  student  must  interact  with  the 
IMI  material,  with  IMI  Level  1  being  the  lowest  and  IMI  Level  4  the  highest.  For  a 
more  complete  definition  of  IMI  levels,  see  Chapter  Three. 


Advanced  Noncommissioned  Officer  Course  (ANCOC)  to  3.5  weeks 
in  residence.  In  addition,  a  new  version  of  the  Basic  Noncommissioned 
Officer  Course  (BNCOC)  Common  Core  used  IMI  and  asynchronous 
DL  methods  to  create  a  flexible  learning  environment  in  which  stu¬ 
dents  can  take  the  course  anywhere  and  anytime  over  a  90-day  period. 

Integrate  TADLP  with  Knowledge  Management  (KM) 

TADLP  has  primarily  been  using  DL  to  support  learning  in  struc¬ 
tured  courses.  But  the  Army  has  started  to  implement  a  construct  in 
which  unstructured  collaborative  learning  and  self-development  also 
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play  a  large  role  in  leader  development  strategies.  As  a  second  area  for 
broadening  TADLP  to  better  support  the  Army’s  training  and  leader 
development  needs,  the  Army  should  pursue  opportunities  to  integrate 
TADLP  with  KM  learning  delivery  programs,  which  involve  the  use 
of  web-based  support  for  soldier  and  leader  learning  outside  the  frame¬ 
work  of  formal  school  courses.  KM  learning  delivery  programs  are 
included  within  the  Center  for  Army  Lessons  Learned  (CALL)  and 
the  Battle  Command  Knowledge  System  (BCKS).  BCKS  includes 
numerous  forums,  including  some  within  TRADOC  and  others  in 
U.S.  Army  Forces  Command.  Both  CALL  and  BCKS  provide  support 
to  leaders,  staff,  and  soldiers  with  access  to  updated  tactics,  techniques, 
and  procedures;  operational  insights  and  other  lessons;  and  a  collabora¬ 
tive  capacity  to  support  short-term  operational  knowledge  needs.  Sol¬ 
diers  can  access  this  support  in  several  ways,  including  online  reposito¬ 
ries,  through  Requests  for  Information  (RFI),  and  community  forums. 

KM  learning  delivery  programs  represent  a  key  set  of  DL  capa¬ 
bilities  to  complement  TADLP’s  support  of  structured  courses.  For  the 
TADLP  TRADOC  Capability  Manager  (TCM),  the  biggest  need  is  to 
establish  a  more  collaborative,  mutually  supportive  effort  between  the 
schools’  instructors  and  training  and  doctrine  developers  on  the  one 
hand,  and  the  CALL  and  BCKS  programs  on  the  other. 

Enhance  Key  Management  Functions  to  Achieve  TADLP 
Transformation 

This  research  identified  important  directions  the  Army  could  take  both 
to  improve  the  IMI-focused  DL  program  and,  more  importantly,  to 
broaden  TADLP  beyond  IMI  to  better  achieve  its  larger  goals.  Gen¬ 
erating  real  movement  in  these  directions  will  involve  a  significant 
change  and  require  a  management  effort  considerably  larger  than  that 
needed  to  keep  the  TADLP  moving  along  its  current  path.  In  addi¬ 
tion  to  the  integration  of  KM  and  TADLP  described  above,  we  have 
identified  four  additional  management  functions  for  improvement  to 
support  these  directional  changes.  Because  the  TADLP  TCM  has  the 
responsibility  to  “develop  and  implement  policies  and  programs  for 
TADLP  throughout  the  Army  training  environment,”  the  TADLP 
TCM  should  be  the  lead  agent  for  improving  these  functions. 
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Institute  a  program-level  evaluation  component.  Key  to  pro¬ 
gram  management  and  improvement  is  the  establishment  of  a  means 
to  evaluate  the  program  to  support  needed  improvements  and  timely 
adjustments.  Our  overall  finding  is  that  no  systematic  evaluation  and 
assessment  program  is  in  place  at  the  program  level.  Clearly,  program 
evaluation  and  assessment  is  a  critical  TCM  function,  not  only  for 
program  management  but  for  making  an  effective  case  for  resources 
and  support.  We  argue  that  this  function  is  one  needing  immediate 
improvement,  and  that  an  effective  system  could  be  established  based 
on  the  methodology  outlined  earlier  in  this  summary. 

Starting  with  the  methods  piloted  by  Arroyo,  TRADOC  could 
begin  to  evaluate  the  impact,  cost-effectiveness,  quality,  and  responsive¬ 
ness  of  courseware.  Our  assessment  of  impact,  cost-effectiveness,  and 
responsiveness  was  accomplished  using  available  data,  and  so  beginning 
the  implementation  would  require  only  the  development  of  a  process  for 
collecting  and  analyzing  available  data. 

Develop  concepts,  plans,  and  directives  for  TADLP  transforma¬ 
tion.  The  TCM’s  office  must  orchestrate  and  support  the  movement 
of  institutional  training  from  a  resident-based  approach  to  one  that 
is  more  DL-based  and  that  includes  learning  outside  the  bounds  of 
structured  courseware.  Increased  coordination,  integration,  and  col¬ 
laboration  across  the  Army  will  be  needed  to  achieve  the  Army’s  DL 
goals.  Central  to  a  successful  transformation  will  be  evaluation  and 
assessment  of  ongoing  TADLP  training,  the  development  of  concepts 
and  plans  for  an  expanded  DL  program,  the  implementation  of  a 
spiral  development  approach  to  achieve  evolutionary  transformation, 
and  increased  representation  of  the  user  in  the  acquisition  of  required 
DL  support  materiel  (e.g.,  DL  classrooms  and  learning  management 
systems).  A  successful  approach  will  also  involve  greater  collaboration 
across  a  wide  range  of  stakeholders,  including  the  proponent  schools 
and  unit  customers. 

Implement  a  spiral  development  approach.  The  use  of  a  spiral 
development  approach  will  be  key  to  implementing  changes  and 
expanding  the  use  of  DL.  Spiral  development  is  a  method  of  rap¬ 
idly  implementing  change  while  simultaneously  allowing  for  ongoing 
assessment  and  improvement.  Under  this  process,  an  initial  version  of 
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a  product  or  program  (such  as  a  DL  course)  is  developed  as  a  kind 
of  work-in-progress.  This  gets  a  working  prototype  fielded  and  piloted 
early,  while  also  allowing  it  to  be  assessed,  refined,  and  revised  in  stages. 
Once  an  improved  version  of  the  product  or  process  is  fielded,  the  pro¬ 
cess  of  assessment  and  revision  continues  in  a  new  cycle. 

Perform  combat  developer  role.  Combat  developer  responsibili¬ 
ties  are  to  specify  requirements  for  materiel  systems  and  to  represent 
the  user  community  through  the  process  of  materiel  development  and 
fielding.  An  important  TCM  role  is  to  serve  as  the  combat  developer 
for  technology  that  supports  TADLP  execution.  Because  the  execution 
of  DL  is  heavily  dependent  on  technological  capabilities,  achieving  an 
expanded  DL  program  is  strongly  related  to  TCM  combat  developer 
success. 

An  examination  of  the  achievements  of  ALMS  indicates  that 
improvement  of  the  TCM’s  performance  in  the  combat  developer  role 
is  needed,  especially  to  generate  improvement  in  utility  and  user  friend¬ 
liness.  ALMS  has  been  a  major  component  of  the  TADLP  for  many 
years,  and,  as  described  earlier,  an  increasing  portion  of  the  TADLP 
budget  has  gone  into  ALMS.  However,  ALMS  achievements  have  been 
limited,  as  evidenced  by  the  small  number  of  proponent  schools  that 
used  the  system. 

The  conclusion  is  that  the  TCM  should  improve  its  ability  to 
define  user  requirements,  obtain  needed  funding,  test  materiel  devel¬ 
oper  products  to  ensure  that  they  meet  user  needs,  and  provide  contin¬ 
ued  general  oversight  of  system  effectiveness  after  fielding. 


Implications 

In  this  report  we  present  options  for  improving  the  existing  TADLP. 
We  make  a  case  for  significantly  expanding  the  use  of  both  structured 
and  unstructured  learning  to  enhance  the  Army’s  leader  training  and 
development  strategies.  While  this  represents  a  significant  shift  from 
existing  practices,  we  think  it  is  needed,  for  DL  can  and  should  play  a 
larger  role  in  the  transformation  of  training  in  the  Army.  In  addition 
to  the  detailed  recommendations  discussed  in  this  report,  top-down 
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command  emphasis,  willingness  to  change,  and  oversight  will  all  be 
key  to  achieving  this  transformation. 
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AAR 

After-Action  Review 

AC 

Active  Component 

ACP 

Army  Campaign  Plan 

ADL 

Advanced  Distributed  Learning 

AIT 

Advanced  Individual  Training 

AKM 

Army  Knowledge  Management 

AKO 

Army  Knowledge  Online 

ALARACT 

All  Army  Activities 

ALC 

Advanced  Leader  Course 

ALMS 

The  Army  Learning  Management  System 

AMC 

Army  Materiel  Command 

AMEDD 

Army  Medical  Department 

ANCOC 

Advanced  Noncommissioned  Officer  Course 

ARFORGEN 

Army  Force  Generation 

ARI 

Army  Research  Institute 

ARNG 

Army  National  Guard 

ASAT 

Automated  Systems  Approach  to  Training 

ASI 

Additional  Skill  Identifier 

ATLDS 

Army  Training  and  Leader  Development  Strategy 

ATSC 

Army  Training  Support  Center 
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ATRRS 

Army  Training  Requirements  and  Resources 
System 

BCC 

Brigade  Coordination  Cell 

BCKS 

Battle  Command  Knowledge  System 

BNCOC 

Basic  Noncommissioned  Officer  Course 

C4ISR 

Command,  Control,  Communications, 
Computers,  Intelligence,  Surveillance,  and 
Reconnaissance 

CAC 

Combined  Arms  Center 

CAC-K 

Combined  Arms  Center— Knowledge 

CAD 
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Introduction 


Distributed  Learning  Is  Becoming  an  Increasingly 
Important  Part  of  the  Army's  Training  Strategy 

The  Army  Distributed  Learning  Program  (TADLP)  is  part  of  the 
Army’s  training  system  and  supports  execution  of  the  Army’s  training 
and  leader  development  strategies.  The  overall  goal  of  these  strategies 
is  to  prepare  soldiers,  leaders,  civilians,  and  organizations  to  conduct 
a  wide  range  of  operational  missions.  Leader  development  strategies 
have  the  additional  goal  of  providing  for  the  long-term  development  of 
Army  leaders.1  Distributed  learning  (DL)  enhances  and  extends  tradi¬ 
tional  methods  of  learning  by  making  use  of  multiple  means  and  tech¬ 
nologies  to  enable  the  delivery  of  training  and  learning  wherever  and 
whenever  soldiers  and  leaders  need  it.  With  the  advancement  of  digital 
technologies,  the  Army  envisions  an  increasingly  important  role  for  DL 
in  all  three  domains  of  Army  soldier  and  leader  training  and  develop¬ 
ment:  institutional  (structured  courses  emanating  from  Army  training 
institutions),  operational  (individual  and  collective  training  conducted 
in  the  operating  force),  and  self-development  (additional  schooling  or 
study  outside  the  institutional  and  operational  domains).2 


1  For  a  description  of  Army  training  and  leader  development  strategies  and  systems,  see 
HQDA,  Army  Regulation  (AR)  350-1,  Army  Training  and  Leader  Development,  August 
2007,  and  HQDA,  Army  Field  Manual  (FM)  7-0,  Training  for  Full  Spectrum  Operations, 
December  2008. 

2  FM  7-0,  Chapter  3. 
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The  demand  for  DL  (i.e.,  the  need  to  be  able  to  train  at  any  time 
and  any  place)  has  increased  since  deployments  began  in  2002.  Posing 
the  largest  training  challenge  to  the  Army  is  the  need  to  prepare  units 
and  leaders  for  a  greater  range  of  possible  operational  missions,  or  full- 
spectrum  operations,  while  at  the  same  time  supporting  a  demanding 
set  of  ongoing  operations.  The  reserve  components  (RC)  have  come  to 
be  employed  as  an  operational  force  rather  than  just  a  strategic  reserve, 
thus  increasing  the  importance  of  a  training  and  leader  development 
system  that  responsively  meets  the  current  needs  of  all  components.  The 
requirement  for  a  full-spectrum  capability  has  expanded  the  number  of 
tasks  that  soldiers  and  leaders  will  be  asked  to  perform,  by  emphasizing 
stability  as  well  as  traditional  combat  tasks.  A  full-spectrum  capability 
also  requires  leaders  who  understand  a  wide  range  of  operational  con¬ 
cepts  and  can  apply  them  across  greatly  varying  conditions. 

These  challenges  are  amplified  because  the  training  system  also 
has  to  cope  with  the  demands  placed  on  it  by  equipment  moderniza¬ 
tion,  modularity,  and  increases  in  the  size  of  the  Army.  The  Army  is 
phasing  in  a  range  of  new  equipment,  including  advanced  capabilities 
in  Command,  Control,  Communications,  Computers,  Intelligence, 
Surveillance,  and  Reconnaissance  (C4ISR).  Fully  leveraging  these 
technologies  requires  training  in  complex  analysis,  planning  real-time 
decisionmaking,  and  rapid  adaptation.  Modular  equipment  concepts 
generally  emphasize  leveraging  C4ISR  systems  and  thus  create  a  greater 
need  to  train  leaders  on  complex  synchronization  skills.3 

The  trends  above  give  rise  to  changes  and  increases  in  the  tasks, 
skills,  and  knowledge  the  Army  needs  to  train  in  its  soldiers  and  lead¬ 
ers.4  Some  areas,  such  as  the  conduct  of  stability  and  counterinsurgency 
operations,  are  now  considered  much  more  important  than  they  were 
in  the  past.  Moreover,  the  tactics  and  techniques  that  underpin  train¬ 
ing  change  as  operational  requirements  and  conditions  change.  Thus, 
the  training  system,  including  DL  courseware,  must  be  highly  respon- 


3  TRADOC,  Army  Comprehensive  Guide  to  Modularity,  October  2004. 

4  ARI,  Training  for  Future  Operations — Digital  Leaders’  Transformation  Insights,  ARI  Spe¬ 
cial  Report  No.  53,  November  2002. 
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sive  so  that  it  can  maintain  currency,  increase  total  learning  delivery 
capability,  and  meet  the  sometimes  differing  needs  of  all  components. 

The  current  operating  environment  and  the  ongoing  high  demand 
for  deployed  units  have  led  the  Army  to  adopt  Army  Force  Generation 
(ARFORGEN),  a  cyclical  readiness  process  that  requires  active  Army 
units  to  move  progressively  through  a  structured  set  of  collective  train¬ 
ing  exercises  over  a  three-year  cycle  and  the  RC  to  go  through  a  five- 
to  six-year  cycle.5  This  cycle  requires  the  basic  individual  skills  and 
tasks  necessary  for  collective  execution  to  be  trained  and  institutional 
courses  to  be  conducted  during  a  fairly  narrow  window  of  time  at  the 
beginning  of  the  cycle,  just  after  a  unit  may  well  be  returning  from  a 
deployment.  Training  and  leader  development  strategies  must  adapt  to 
focus  on  supporting  unit  readiness  in  the  context  of  this  cycle.  Thus, 
the  requirement  places  great  demand  on  the  scheduling  of  institutional 
courses  and  limits  the  length  of  time  soldiers  have  to  spend  at  a  school- 
house  (away  from  home  station). 

At  the  same  time  that  demands  on  the  training  system  are  increas¬ 
ing,  the  overall  trend  has  been  to  reduce  the  amount  of  training  con¬ 
ducted  and  required  at  the  schoolhouse.6  To  provide  the  training  elimi¬ 
nated  by  these  reductions  and  cover  the  new  required  skills  and  tasks, 
the  Army  has  aimed  to  leverage  DL  methods  and  the  potential  of  the 
self-development  domain. 

Plans  to  employ  DL  are  documented  in  multiple  Army  publi¬ 
cations.  The  Army  Campaign  Plan  (ACP),  which  directs  planning, 
preparation,  and  execution  of  Army  transformation,  has  “Train  the 
Army  and  grow  adaptive  leaders”  as  one  of  its  seven  major  goals  and 
identifies  DL  as  a  key  means  of  supporting  its  achievement.7  Further, 
the  ACP  outlines  a  major  role  for  DL  in  supporting  a  life-long  learn- 


5  For  more  details,  see  Annex  F  (ARFORGEN  Implementation  Plan)  and  Annex  G  (Life- 
cycle  Management)  in  HQDA,  Army  Campaign  Plan— Change  5,  April  2007. 

h  Initial  Military  Training  courses  are  the  exception,  where  enlisted  and  officer  courses 
have  lengthened  by  about  a  week  to  increase  combat  skills  training  (Warrior  tasks). 

7  Army  Campaign  Plan— Change  5,  April  2007.  Annex  L  outlines  the  specifics  of  Army 
training  and  leader  development  transformation  plans  and  requirements.  The  ACP  is  contin¬ 
uously  being  revised,  but  a  newer  draft  version  we  examined  retains  the  objective  of  growing 
adaptive  leaders  and  outlines  DL  as  a  key  means  of  supporting  its  achievement. 
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ing  construct.  Under  this  construct,  the  Army’s  institutional  doctrine 
and  leader  development  capability  will  transform  to  create  “classrooms 
without  walls,”  allowing  access  to  knowledge  and  expertise  from  units’ 
home  stations  and  deployed  locations.  Supporting  this  concept  are  a 
number  of  initiatives  in  the  TRADOC  Campaign  Plan  (TCP),  includ¬ 
ing  the  goal  of  establishing  an  exportable  training  capability  for  the 
Noncommissioned  Officer  Education  System  (NCOES).8 

Although  the  effort  to  transform  training  and  leader  development 
programs  is  just  beginning,9  some  revisions  are  already  under  way.  A 
leading  effort  in  this  regard  has  been  to  transform  NCOES,  and  a 
major  role  for  DL  is  envisioned.  The  current  direction  is  to  place  most 
common  core  instruction  into  required  DL  modules,  called  Structured 
Self-Development  (SSD).10  Each  SSD  module  is  envisioned  as  being  80 
hours  in  length.  SSD1  will  precede  the  Warrior  Leader  Course,  SSD2 
the  Advanced  Leader  Course  (ALC),  SSD3  the  Senior  Leader  Course 
(SLC),  and  SSD4  the  Sergeants  Major  Course.  SSD5  will  be  after  the 
Sergeants  Major  Course.  As  a  part  of  the  NCOES  transformation  plan, 


8  U.S.  Army  Training  and  Doctrine  Command,  TRADOC  Campaign  Plan  with  Change  1, 
June  2006. 

9  The  Army  has  developed  a  draft  “Army  Training  and  Leader  Development  Strategy” 
(ATLDS),  which  outlines  broad  goals  and  objectives  but  has  not  yet  been  refined  to  outline 
a  specific  concept  or  plan  on  how  these  are  to  be  achieved.  See  HQDA,  Army  Training  and 
Leader  Development  Strategy,  draft,  March  2008. 

10  We  have  reviewed  numerous  briefings  on  NCOES  transformation  and  have  discussed 
this  program  in  visits  with  staff  members  at  TRADOC  Headquarters,  the  U.S.  Army  Ser¬ 
geants  Major  Academy,  and  the  U.S.  Army  Armor  School.  Under  NCOES  transformation, 
the  names  and  objectives  of  the  NCOES  courses  for  E6s  and  E7s  changed  in  October  2009. 
The  course  for  E6s  previously  titled  the  “Basic  NCO  Course”  (BNCOC)  had  the  objective 
of  teaching  E6-level  tasks  and  skills.  That  course  is  now  the  ALC;  under  the  transformed 
NCOES  program,  the  goal  is  still  to  have  NCOs  attend  the  ALC  prior  to  promotion  to  E6, 
but  they  will  learn  E6-  and  some  E7-level  tasks  and  skills.  The  course  for  E7s  was  previously 
titled  the  “Advanced  NCO  Course”  (ANCOC)  and  taught  E7  tasks  and  skills.  That  course 
is  now  the  SLC;  under  the  transformed  NCOES  program,  the  goal  is  still  that  NCOs  will 
attend  SLC  prior  to  promotion  to  E7,  but  they  will  learn  E7  and  some  First  Sergeant  tasks 
and  skills.  It  should  be  noted  that  current  promotion  policies  allow  for  promotion  to  E6 
without  ALC  and  to  E7  without  SLC,  but  these  policies  do  not  align  with  NCOES  transfor¬ 
mation  goals,  nor  with  the  design  of  NCOES  courses.  For  the  remainder  of  the  report,  we 
will  use  the  current  course  titles  except  when  specifically  discussing  NCOES  courses  prior  to 
2009. 
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DL  will  also  be  used  to  provide  Guided  Self-Development  (GSD),  com¬ 
prised  of  a  set  of  recommended  but  optional  learning  modules.  A  key 
purpose  of  SSD,  and  to  some  extent  GSD,  is  to  allow  for  a  combina¬ 
tion  of  expanded  NCOES  content  and  reduced  resident  course  length. 


TADLP  Resources  Are  Declining 

Although  the  need  for  DL  appears  to  be  increasing,  resources  for 
TADLP  declined  sharply  in  the  years  leading  up  to  this  assessment.  In 
the  President’s  Budget  released  in  Lebruary  2005,  the  Army  projected 
it  would  spend  over  $1  billion  on  TADLP  from  LY  2006  through  LY 
2011,  an  amount  that  represented  a  gradual  increase  in  the  program’s 
overall  funding  level  over  the  period  and  over  the  levels  seen  in  the  first 
half  of  the  decade  (see  Figure  1.1).  However,  three  years  later,  in  Janu¬ 
ary  2008,  the  Army  was  funding  the  program  at  a  level  that  was  40 
percent  less  than  that  earlier  amount  over  those  same  years.11  Figure  1.1 
shows  that  actual  funding  for  FY  2006  and  FY  2007  was  more  than  30 
percent  lower  than  the  former  projections  and  that  investments  from 
that  point  onward  would  go  down  even  further.  Moreover,  an  even 
smaller  proportion  of  the  total  dollar  amounts  is  budgeted  to  go  to  U.S. 
Army  Training  and  Doctrine  Command  (TRADOC)  relative  to  other 
commands,  indicating  that  funding  for  TRADOC  is  decreasing  faster 
than  the  funding  for  those  other  commands. 

Budget  figures  also  reveal  that  the  production  of  DL  content  is 
being  de-emphasized  within  TADLP.  The  main  components  of  TADLP 
are  (a)  courseware  development,  (b)  the  Army  Learning  Management 
System  (ALMS),  and  (c)  DL  classrooms,  both  fixed  and  deployable.12 
The  production  of  courseware  should  be  the  centerpiece  of  the  pro¬ 
gram;  the  other  two  elements  are  supporting  components.  However, 


11  Derived  from  a  comparison  of  the  President’s  Budget  (PB)  0607  lock  (released  on  Febru- 
ary  16,  2005),  with  PB09  lock  (released  on  January  28,  2008)  in  the  PROBE  database,  where 
the  Army  stores  budget-related  data. 

12  There  are  a  number  of  lesser  components  of  TADLP  not  covered  in  this  report,  including 
the  Army  e-Learning  program,  which  delivers  off-the-shelf  commercial  DL  courseware  to 
provide  information  technology,  foreign  language,  and  other  types  of  workforce  training. 
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Figure  1.1 

Army  Resources  Dedicated  to  TADLP  (FY  2006-2011) 


we  found  that  courseware  constituted  a  relatively  small  and  decreasing 
component  of  TADLP  funding.  For  example,  over  FY  2003  through 
FY  2009,  courseware  funding  was  32  percent  of  total  program  fund¬ 
ing,  and  that  share  was  projected  to  go  down  to  an  average  of  only  23 
percent  over  FY  2010  through  2013. 13 


Purposes  of  This  Study 

Considering  the  growing  importance  of  DL  within  the  Army’s  train¬ 
ing  as  well  as  the  increasingly  constrained  resources  available  for  DL, 
TRADOC  asked  RAND  Arroyo  Center  to  assess  the  performance 
of  TADLP  in  the  FY  2007-2008  time  frame.  A  key  purpose  of  this 
assessment  was  to  document  the  state  of  TADLP  at  that  time  in  order 
to  establish  a  baseline  against  which  future  improvements  to  the  pro- 


13  Derived  from  PB09  lock  (released  on  January  28,  2008),  plus  final  funding  figures  for  the 
years  before  FY  2006. 
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gram  could  be  measured.14  In  addition,  the  project  sought  to  propose 
options  that  the  Army  could  implement  to  improve  DL  performance. 
This  second  objective  had  two  major  purposes.  The  first  year  of  the 
project  focused  on  improvements  to  TRADOC’s  program  for  devel¬ 
oping  computer-based  interactive  multimedia  instruction  (IMI).15  The 
second  year  of  the  project  focused  on  the  potential  for  making  strate¬ 
gic  improvements  to  TADLP  so  that  the  Army  can  leverage  DL  more 
effectively  in  the  future.  The  three  key  purposes  of  the  study  are  dis¬ 
played  in  Figure  1.2. 


Approach 

The  study  was  done  in  three  stages,  one  for  each  of  the  key  purposes 
as  shown  in  Figure  1.2.  In  the  first  stage,  we  used  data  from  FY  2006 
to  assess  TADLP  courseware  against  five  measures  of  effectiveness: 
impact  on  training,  efficiency  of  process,  quality  of  content,  cycle 
time  of  production,  and  responsiveness  to  the  need  for  change.  In  the 
second  stage,  we  developed  options  for  improving  the  IMI  program  (as 
it  existed  up  through  2008)  to  address  the  areas  of  weakness  identified 
in  the  first  stage.  Finally,  in  the  third  stage,  we  developed  and  outlined 
broader  options  for  increasing  the  program’s  impact  and  improving 
cost-effectiveness  over  the  longer  term. 


14  The  report  does  not  cover  enhancements  or  changes  to  TADLP  after  the  assessment 
period.  However,  in  the  case  of  the  knowledge  management  (KM)  program,  a  more  recent 
assessment  has  allowed  us  to  update  our  comments  to  the  end  of  FY  2010.  See  Chapter  Six. 

15  As  per  TRADOC  Regulation  350-70,  IMI  is  a  term  applied  to  a  group  of  predominantly 
interactive,  electronically  delivered  training  and  training  support  products.  IMI  can  link  a 
combination  of  media,  to  include  but  not  be  limited  to  programmed  instruction,  video  tapes, 
slides,  film,  television,  text,  graphics,  digital  audio,  animation,  and  up-to-full  motion  video, 
to  enhance  the  learning  process.  IMI  products  include  instructional  software  and  software 
management  tools  used  in  support  of  instructional  programs.  IMI  products  are  teaching 
and  management  tools  and  may  be  used  in  combination  or  individually.  Used  individually, 
not  all  IMI  products  can  be  considered  interactive,  multimedia,  or  instructional.  However, 
IMI  products,  when  used  in  combination  with  one  another,  are  interactive,  multimedia,  and 
instructional. 
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Figure  1.2 

Conceptual  Design  of  Project 


Baseline  FY06  Improvements 


RAND  MG1016-1.2 


We  focused  the  majority  of  our  analysis  on  issues  related  to  DL 
courseware.  This  decision  was  supported  by  early  findings  confirming 
that  only  a  small  amount  of  high-priority  courseware  was  active  in  FY 
2006.  Clearly  the  “long  pole  in  the  tent”  for  the  program,  the  produc¬ 
tion  of  more  content  would  be  needed  before  the  other  two  parts  of 
TADLP  (learning  management  infrastructure  and  facilities)  would  be 
relevant  to  the  Army’s  key  requirements  for  DL. 

We  further  focused  our  efforts  within  the  area  of  courseware  on 
Army  training  most  directly  connected  to  readiness  (and  also  the  lon¬ 
gest  and  most  demanding  training),  the  Army’s  “priority  one”  courses. 
These  included  military  occupational  specialty  (MOS)  transformation 
courses  supporting  reserve  component  duty  MOS  qualification  needs, 
key  Professional  Military  Education  courses — the  Basic  and  Advanced 
Noncommissioned  Officer  Courses  (BNCOCs  and  ANCOCs)  and 
Captains  Career  Courses16 — and  the  most  important  functional 
courses:  those  important  enough  to  units  to  be  designated  as  additional 
skill  identifiers  or  special  qualifications  identifiers. 

To  conduct  our  DL  impact  analyses  (stage  one),  we  used  data¬ 
bases  maintained  by  Army  G-l  (the  Army  Training  Requirements  and 


16  We  found  only  a  smattering  of  Officer  Education  System  (OES)  courses  that  used  DL 
below  the  Captain  level.  While  some  OES  courses  for  the  Major  level  and  above  use  distrib¬ 
uted  means,  they  are  not  funded  through  TADLP  and  thus  are  excluded  from  this  report. 


Introduction  9 


Resources  System,  or  ATRRS),  Army  Training  Support  Center  (vari¬ 
ous  data  on  proposed  and  funded  courses),  and  the  Training  Opera¬ 
tions  Management  Activity  (DL  course  graduates  over  time).  To  iden¬ 
tify  the  characteristics  of  active  DL  courses,  we  also  surveyed  and 
interviewed  representatives  from  Army  proponent  schools,  contractors 
and  headquarters  staff  from  Army  Training  Support  Center  (ATSC) 
and  TRADOC,  and  various  other  DL  experts  inside  and  outside  the 
Army  and  the  military. 

As  part  of  this  effort,  a  survey  of  seven  questions  was  sent  to 
TRADOC  proponent  schools  with  active  DL  courses.  The  survey 
focused  on  each  piece  of  courseware  that  fell  into  the  project’s  area  of 
focus  (those  courses  tied  most  directly  to  readiness).17  The  questions 
can  be  found  in  Appendix  A. 

The  first  stage  also  included  a  quality  review  of  selected  IMI 
courseware.  We  designed  and  conducted  a  limited  qualitative  evalu¬ 
ation  of  a  stratified  sample  of  the  instructional  design  of  10  TADLP 
courses  developed  between  2005  and  2007  (focusing  on  nearly  80  les¬ 
sons  and  190  hours  of  instruction).  Criteria  for  the  evaluation  encom¬ 
passed  detailed  technical,  production-quality,  and  pedagogical  ele¬ 
ments  of  the  courses. 

We  employed  a  number  of  other  approaches  to  gain  insights  on 
how  TADLP  could  be  improved  (stages  two  and  three).  We  drew  upon 
various  data  sources  for  this  part  of  the  analysis: 

•  Focus  groups  with  three  different  stakeholder  groups  attending 
TRADOC’s  annual  DL  conference  in  March  2007:  (a)  TADLP 
contractors  for  custom  content  development;  (b)  ATSC  course¬ 
ware  managers  and  contracting  officers  representatives  (CORs); 
and  (c)  Army  proponent  school  representatives  attending  the 
conference. 

•  A  comparative  review  of  courseware  production  processes  and 
practices  within  TRADOC  and  within  the  commercial  sector. 


17  A  currently  active  course  was  defined  as  one  which,  according  to  ATRRS,  had  enroll¬ 
ments  in  FY  2006  or  (through  the  first  half  of)  FY  2007,  or  which  had  requirements  in  FY 
2007  or  FY  2008. 
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This  included  interviews  with  staff  at  ATSC  as  well  as  production 
managers  at  a  number  of  commercial  training  organizations,  and 
experts  from  DL  industry  groups. 

•  Structured  interviews  with  TRADOC  proponent  schools.  We 
conducted  extended  telephone  interviews  with  personnel  associ¬ 
ated  with  20  Army  DL  programs  that  applied  for  and  received 
TADLP  funding  in  the  past.  Participants  typically  included 
contracting  representatives,  course  managers,  team  leads,  train¬ 
ing  division  or  branch  chiefs  responsible  for  the  production  of 
DL,  and  in  some  cases  the  school’s  director  of  training.  Topics 
addressed  during  the  interview  included  the  role  of  DL  in  the 
school’s  larger  training  strategy,  how  the  schools  selected  training 
content  to  be  converted  to  DL,  the  amount  of  resources  dedicated 
to  the  DL  program  at  the  school,  school  efforts  to  assess  the  DL 
products  they  produced,  and  obstacles  and  suggested  improve¬ 
ments  related  to  the  implementation  of  TADLP. 


Organization  of  This  Report 

The  remainder  of  this  report  is  divided  into  seven  chapters. 

•  Chapter  Two  provides  an  overview  of  TADLP. 

•  Chapter  Three  provides  the  results  of  the  first  stage  of  the  analy¬ 
sis,  an  assessment  of  the  status  of  Army  DL  as  of  FY  2006  with 
regard  to  courseware  impact,  efficiency,  quality,  cycle  time,  and 
responsiveness. 

•  Chapter  Four  provides  the  results  of  the  second  stage  of  the  analy¬ 
sis,  which  identifies  potential  improvements  to  the  Army’s  IMI 
program  that  can  positively  influence  TADLP  outcomes  with 
regard  to  the  measures  of  effectiveness  presented  in  Chapter  Three. 

•  Chapter  Five  is  the  first  of  three  chapters  dealing  with  the  results 
of  the  third  stage  of  the  analysis,  which  focuses  on  broader  changes 
the  Army  might  make  in  its  approach  to  DL. 

•  Chapter  Six  presents  the  next  set  of  third  stage  results  by  exam¬ 
ining  the  Army’s  use  of  knowledge  management  (KM)  methods 
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to  deliver  learning,  assessing  the  benefits  thereof,  and  considering 
how  KM  learning  delivery  might  be  integrated  into  DL. 

•  Chapter  Seven  discusses  the  management  functions  that  are  key 
to  improving  and  broadening  TADLP,  and  suggests  directions  for 
their  improvement. 

•  Chapter  Eight  summarizes  the  major  conclusions,  recommenda¬ 
tions,  and  implications  of  this  study. 

The  report  also  contains  two  appendixes.  Appendix  A  describes 
the  survey  and  interview  methods  used  and  provides  the  two  question¬ 
naires,  one  used  to  determine  the  specific  characteristics  of  DL  courses, 
and  another  used  to  guide  semi-structured  interviews  with  proponent 
schools.  Appendix  B  discusses  in  greater  detail  the  results  of  interviews 
with  training  development  personnel  from  20  Army  schools  and  pro¬ 
grams;  results  from  these  interviews  were  used  as  one  input  for  recom¬ 
mended  improvements  presented  in  Chapter  Four. 


CHAPTER  TWO 


Features  of  The  Army  Distributed  Learning 
Program  (TADLP) 


In  this  chapter  we  provide  an  overview  of  TADLP  as  it  existed  in  the 
FY  2006-2008  time  frame.  While  many  organizations  in  the  Army 
develop  and  deliver  DL,1  TADLP  is  the  only  formal  program  in  the 
Army  dedicated  to  DL.  A  basic  understanding  of  the  program  forms 
the  context  for  our  assessment  and  recommendations. 

There  are  four  sections  in  this  chapter.  First,  we  review  the  fund¬ 
ing,  organization,  and  main  components  of  TADLP.  Second,  we 
examine  the  Army  Learning  Management  System  (ALMS),  which 
TRADOC  has  developed  to  manage  DL  and  support  resident  instruc¬ 
tion.  Third,  we  briefly  describe  the  network  of  worldwide  facilities  sup¬ 
ported  by  the  DL  program.  Finally,  and  consistent  with  our  empha¬ 
sis  on  IMI  in  this  report,  we  provide  a  description  of  the  process  for 
IMI  production.  The  discussion  is  included  as  a  “feature”  of  TADLP 
because  it  will  provide  important  background  for  a  number  of  our  sub¬ 
sequent  recommendations. 


1  For  example,  much  IMI  is  produced  by  program  managers  (PMs)  in  the  Army  to  sup¬ 
port  training  in  connection  with  the  fielding  of  materiel  systems.  Army  schools  and  com¬ 
mands  also  sometimes  purchase  their  own  infrastructure  and  individually  produce  their 
own  courseware  either  by  themselves  or  in  partnership  with  commercial  or  academic 
partners. 


13 


14  Making  Improvements  to  The  Army  Distributed  Learning  Program 


TADLP  Overview 

Program  Components  and  Budgets 

With  an  overall  budget  in  FY  2006  of  about  $125  million  per  year, 
TADLP  is  funded  by  the  Training  PEG  (Program  Evaluation  Group) 
and  pursues  an  expansive  set  of  DL  and  related  objectives  for  Army 
training.2  The  main  components  of  TADLP  support  Army-unique 
training  with  courseware  development,  the  ALMS,  and  DL  class¬ 
rooms,  both  fixed  and  deployable.3  Three  Army  organizations  receive 
the  majority  of  TADLP  funds:  TRADOC,  Army  National  Guard 
(ARNG),  and  the  Program  Manager  for  Distributed  Learning  Systems 
(PM  DLS).4  PM  DLS  receives  about  half  of  all  funds. 

In  FY  2006,  funding  for  TADLP  was  allocated  approximately  as 
follows:  25  percent  for  courseware,  25  percent  for  learning  manage¬ 
ment  systems,  and  the  remaining  50  percent  for  facilities  (e.g.,  digi¬ 
tal  training  centers).  TRADOC  and  ARNG  receive  the  courseware 
funds,  PM  DLS  receives  the  funds  for  learning  management  systems, 
and  all  three  organizations  receive  the  funds  for  facilities.  While  the 
largest  component  of  funding  goes  toward  facilities,  the  numbers  are 
somewhat  misleading  from  a  DL  perspective  because  those  facilities 
provide  for  a  wide  range  of  uses  related  to  digital  communication,  and 
thus  serve  to  provide  much  more  than  “support  of  DL.” 

As  briefly  discussed  in  the  last  chapter,  budget  trends  show  a 
decreasing  emphasis  on  TADLP.  After  generally  increasing  through 
the  middle  of  FY  2005,  budgeted  funding  for  TADLP  decreased  in  FY 
2006  and  for  the  future.  Further,  the  downward  turn  has  been  sharper 


2  In  the  application  of  distributed  learning  methods,  TADLP  supports  the  “DoD  intent  to 
deliver  ‘learner  centric’  quality  training  when  and  where  required,  increasing  and  sustaining 
readiness  throughout  the  force,  Active  and  Reserve.” 

3  As  previously  noted,  there  are  a  number  of  lesser  components  of  TADLP  not  covered  in 
this  report,  including  the  Army  e-Learning  program. 

4  DLS  is  part  of  the  U.S.  Army  Program  Executive  Office  Enterprise  Information  Systems 
(PEO  EIS).  PEO  EIS  is  responsible  for  project  management  of  DoD  and  Army  business  and 
combat  service  support  systems,  as  well  as  related  Army  communication  and  computer  infra¬ 
structure.  Although  most  PMs  are  funded  out  of  the  equipping  PEG,  DLS  is  funded  out  of 
training,  given  its  importance  there. 
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in  recent  years.  For  example,  as  of  February  2005  the  Army  expected  to 
spend  more  than  $1  billion  dollars  on  TADLP  from  FY  2006  through 
FY2011.  But  three  years  later,  in  January  2008,  the  Army  had  provided 
40  percent  less  funding  than  previously  planned  over  those  years.5 

In  addition,  a  smaller  proportion  of  funds  is  going  to  TRADOC 
relative  to  the  proportion  going  to  the  other  major  DL  claimants.  PM 
DLS  appears  to  be  receiving  a  relatively  steady  steam  of  funds,  but 
TRADOC  and  the  ARNG  experienced  a  significant  decrease.  More¬ 
over,  TRADOC  funds  are  declining  at  a  greater  rate  than  ARNG 
funds. 

Budget  trends  also  show  that,  in  the  future,  TADLP  expenditures 
for  courseware  are  expected  to  account  for  a  decreasing  percentage  of 
total  funding.  For  example,  in  FY  2003-2009,  courseware  accounted 
for  an  average  of  32  percent  of  total  TADLP  funding.  In  contrast,  in 
FY  2010-2013,  courseware  funding  amounts  to  only  23  percent  of 
total  TADLP  funding,  a  reduction  of  approximately  one-third.  More¬ 
over,  funding  for  courseware  is  increasingly  going  to  the  ARNG  rather 
than  TRADOC.  Prior  to  FY  2006,  nearly  all  courseware  funding 
went  to  TRADOC,  but  in  FY  2006-2011,  40  percent  of  the  funds  are 
allocated  to  the  ARNG,  with  that  percentage  increasing  in  the  later 
years.  This  trend  reflects,  in  part,  increasing  specialization  in  course¬ 
ware  development,  as  the  ARNG  focuses  more  on  RC  needs  and  the 
reclassification  mission,  and  TRADOC  specializes  in  active  compo¬ 
nent  (AC)  needs  and  NCOES.  The  trend  may  also  reflect  a  Depart¬ 
ment  of  the  Army  (DA)  perception  of  a  more  efficient  ARNG  process 
for  developing  DL  courseware. 

TADLP  also  depends  critically  on  support  resources  from 
TRADOC  and  the  Installation  Management  Command.  These 
resources,  which  are  not  tracked  or  attributed  to  DL,  include  both 
manpower  (e.g.,  training  development  expertise,  subject  matter  experts 
to  support  contractors,  and  program  management  resources)  and  dol¬ 
lars  (e.g.,  to  purchase  installation  and  information  services). 


5  Derived  from  a  comparison  of  PB0607  lock  (released  on  February  16,  2005),  with  PB09 
lock  (released  on  January  28,  2008)  in  the  PROBE  database. 
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Players  and  Roles 

TADLP  is  managed  overall  by  DA’s  G-3/5/7.6  Design  and  implementa¬ 
tion  responsibilities  have  been  given  to  the  TRADOC  Capability  Man¬ 
ager  (TCM),  the  head  of  the  Distributed  Learning  Directorate  within 
TRADOC’s  G-3/5/7.  The  TCM  has  the  TRADOC  staff  responsibility 
to  “develop  and  implement  policies,  plans,  and  programs  for  TADLP 
throughout  the  Army  training  environment.”7 

One  of  the  key  roles  of  the  TCM  is  as  the  “combat  developer” 
for  DL.  The  primary  duties  of  the  combat  developer  are  to  specify 
requirements  for  materiel  systems  and  to  represent  the  user  community 
through  the  materiel  development  process.  In  the  case  of  DL,  materiel 
systems  would  include  learning  management  systems  (LMSs)  and  DL 
facilities.  PM  DLS  is  the  materiel  developer  for  DL.  The  materiel  devel¬ 
oper  is  responsible  for  the  acquisition  and  management  of  materiel  sys¬ 
tems  (i.e.,  LMSs  and  facilities)  that  meet  the  requirements  specified  by 
the  combat  developer.8 

The  TCM  for  TADLP  has  also  recently  had  its  DL  development 
responsibilities  expanded  to  include  the  establishment  of  an  Army¬ 
wide  process  for  DL  governance.  This  role  includes  establishing  over¬ 
sight  over  the  development,  management,  registration,  and  delivery  of 
DL  courseware  not  only  within  TADLP  but  throughout  the  Army.9 

The  Army  Training  Support  Center  (ATSC)  is  a  field  operating 
activity  under  TRADOC  that  provides  management  support  in  the 
execution  of  courseware  development  contracts  and  in  courseware  test¬ 
ing,  and  in  the  development  of  DL  technical  standards  and  specifi¬ 
cations.  Schools  and  other  training  organizations  are  responsible  for 
designing,  developing,  and  delivering  DL  to  the  ultimate  customers, 
who  are  Army  soldiers,  leaders,  and  unit  commanders. 


7  Other  staff  elements  of  DA  are  also  involved,  especially  the  G-l,  which  has  primary 
responsibility  for  personnel  readiness. 

7  TRADOC  Regulation  10-5-1. 

8  The  integration  of  the  roles  of  the  combat  developer  and  materiel  developer  is  complex;  it 
is  governed  by  statute  as  well  as  policy  and  regulation.  See  AR  70-1,  Army  Acquisition  Policy, 
December  2003,  and  AR  71-9,  Materiel  Requirements. 

9  DA  message  on  “All  Army  Activities”  (ALARACT),  May  6,  2007. 
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Courseware 

We  now  discuss  DL  courseware  produced  under  TADLP.  DL  course¬ 
ware  is  defined  broadly  by  the  Army  as  including  all  instructional 
media  using  a  technology  interface.  DL  courseware  includes  IMI,  cor¬ 
respondence  courses  (if  delivered  digitally),  simulation  and  gaming 
when  used  within  instructional  media,  and  video  teletraining  (VTT) 
and  other  forms  of  collaborative  training.10 

Courseware  Development  Goals 

The  goal  for  TADLP,  established  at  the  program’s  inception  in  1996, 
has  been  to  produce  525  completed  DL  courses  by  2010.  From  its  incep¬ 
tion  through  mid-FY  2006,  over  400  DL  courses  had  been  funded  for 
development  under  TRADOC.  Scores  of  Army  proponent  schools, 
commands,  and  other  agencies  have  successfully  participated  in  DL 
courseware  development.  The  organizations  with  the  most  courseware 
completions  (or  expected  completions)  are  the  following  centers  and 
schools:  Army  Medical  Department  (AMEDD),  JFK  Special  Warfare, 
Ordnance,  Military  Intelligence,  Transportation,  Engineer,  Quarter¬ 
master,  Field  Artillery,  Signal,  Chemical,  and  Armor.  These  11  account 
for  70  percent  of  the  completions. 

Courseware  Prioritization 

The  choice  of  which  DL  courses  to  fund  is  determined  by  a  prioritiza¬ 
tion  process  in  which  schools  (and  other  commands  and  agencies)  pro¬ 
pose  content  to  be  converted,  and  TRADOC  determines  priorities  for 
funding.  Prioritization  within  training  is  based  on  a  number  of  consid¬ 
erations,  including  the  following: 

•  Mission  criticality  and  emerging  Army  training  priorities  (e.g.,  as 
reflected  in  commanding  general  or  proponent  school  priorities). 

•  Suitability  of  content  for  DL  delivery,  including  the  likelihood  of 
content  to  change. 


10  Among  the  exclusions  to  this  broad  definition  of  “DL  courseware”  specifically  cited  by  the 
Army  are  simulators  embedded  in  actual  or  virtual  equipment. 


18  Making  Improvements  to  The  Army  Distributed  Learning  Program 


•  Return  on  investment  (ROI)  considerations  (e.g.,  student  load, 
number  of  hours). 

•  Performance  of  school  in  past  development  processes  (i.e.,  fielding 
rate,  submission  of  documentation). 

•  Equitable  distribution  of  DL  among  schools. 


DL  Courses  Active  During  FY  2006 

To  help  ourselves  understand  the  nature  of  the  Army’s  program,  we 
looked  more  closely  into  active  DL  courses  that  had  the  highest  Army 
priority.  In  line  with  our  focus  on  FY  2006  as  a  base  year,  we  defined 
courses  to  be  “active”  if  ATRRS  showed  enrollments  during  FY  2006. 
The  course  categories  of  highest  priority  included  MOS  reclassification 
courses  and  selected  Professional  Military  Education  courses  supported 
byTADLP,  including  Basic  NCO  courses  (BNCOCs),  Advanced  NCO 
courses  (ANCOCs),  Captains  Career  Courses  (CCCs)  for  RC  soldiers, 
and  key  functional  courses  designed  as  “additional  skill  identifiers” 
(ASIs)  or  “special  qualifications  identifiers”  (SQIs).11  Those  courses 
accounted  for  60  percent  of  all  TADLP-funded  courses  through  FY 
2006.  We  identified  103  active  DL  courses  in  FY  2006  out  of  227 
funded.  The  most  prevalent  were  reclassification  courses,  and  the  least 
prevalent  were  ANCOC  courses. 

DL  Course  Characteristics 

For  each  active  course,  we  asked  the  proponent  to  identify  the  course 
characteristics  and  to  describe  how  it  was  used  in  practice.  Below  we 
describe  the  salient  features  of  high-priority  DL  courses  in  FY  2006. 

Asynchronous  IMI  was  the  main  DL  modality.  Although  DL 
is  defined  broadly  in  regulations  as  including  many  forms  of  instruc¬ 
tional  media,  the  active,  high-priority  courses  used  almost  exclusively 
asynchronous  IMI,  that  is,  DL  in  which  the  learning  content  is  deliv¬ 
ered  by  the  software  without  the  need  of  an  instructor,  thereby  provid- 


11  While  the  NCOES  courses  after  October  2009  are  named  ALC  and  SLC,  in  this  chapter 
we  are  referring  to  courses  taught  in  2006-2008,  and  we  will  use  BNCOC  and  ANCOC, 
their  names  during  that  period. 
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ing  the  student  with  the  flexibility  to  access  instruction  at  any  time.  A 
few  courses  used  VTT,  a  form  of  collaborative  media,  including  two 
courses  (Battle  Staff  Course  and  BNCOC  Common  Core)  that  gener¬ 
ate  some  of  the  highest  enrollments  in  the  program,  especially  for  AC 
soldiers. 

DL  was  typically  used  as  the  initial  segment  of  a  phased  learn¬ 
ing  model.  Most  Army  DL  courses  are  not  courses  at  all,  but  modules 
that  support  a  phased  learning  strategy  for  a  larger  course.  Most  often 
the  DL  phase  served  as  cognitive  preparation  for  the  longer  residential 
portion  of  the  course.12  Moreover,  most  of  that  preparation  was  aimed 
at  the  “knowledge,”  and  “comprehension”  levels  of  learning  in  Bloom’s 
taxonomy  of  learning.13  The  “application”  and  higher  levels  of  learn¬ 
ing  needed  for  true  competence  in  leader  tasks  were  almost  exclusively 
left  to  the  residential  portion  of  the  course,  even  though  DL  has  some 
capabilities  in  those  areas.14 

Army  DL  was  primarily  self-paced  learning,  with  little  student- 
instructor  interaction.  Although  the  Army  definition  of  DL  allows 
for  both  synchronous  and  asynchronous  interaction  between  student 
and  instructor,  most  Army  IMI  in  FY  2006  was  self-paced,  with  most 
interaction  taking  place  between  student  and  media.15  In  our  survey  of 
active  DL  courses,  we  found  that  78  percent  were  described  as  having 
“very  low”  student-instructor  interaction,  and  another  10  percent  as 
having  “low”  interaction.  At  a  minimum,  students  would  have  the 
Army  Help  Desk  as  a  number  to  call  for  support,  which  could  forward 
substantive  questions  to  proponent  schools  for  a  response.  In  addition, 
in  at  least  one  case,  the  proponent  school  reported  receiving  some  calls 


12  In  some  cases,  the  DL  phases  also  cover  common  core  material  that  is  not  taught  in  resi¬ 
dence  phases. 

13  Ninety-two  percent  of  the  DL  modules  were  tested  course  requirements,  with  the  remain¬ 
der  used  as  self-study  or  homework. 

14  For  further  explanation,  see  http://www.nwlink.com/-donclark/hrd/bloom.html.  The 
levels  of  learning  in  Bloom’s  cognitive  domain,  in  order  of  increasing  complexity  of  the 
learning  goal,  are  knowledge  (being  able  to  recall  information  from  memory),  comprehen¬ 
sion,  application,  analysis,  synthesis,  and  evaluation. 

15  Varying  levels  of  interactivity  between  learner  and  media  are  defined  in  Chapter  Three. 


20  Making  Improvements  to  The  Army  Distributed  Learning  Program 


directly  from  learners  of  DL  courses,  although  it  did  not  have  anyone 
dedicated  to  the  task  of  student  support. 

DL  course  phases  were  relatively  long  for  self-paced  IMI.  Most 
DL  modules  required  weeks  of  effort  for  learners  to  complete.  Course 
modules  varied  from  12  to  420  academic  hours  in  length,  with  the 
average  being  about  60  hours  for  all  types  of  high-priority  courses 
except  CCC,  which  averaged  about  twice  that  length. 

DL  course  modules  focused  most  on  RC  readiness.  The  list 
of  DL  courses  active  in  FY  2006  showed  TADLP’s  emphasis  on  RC 
readiness.  Both  the  MOS  reclassification  and  CCC  course  modules, 
accounting  for  over  half  of  the  total,  had  almost  exclusively  an  RC 
learner  base.  Overall,  schools  reported  that  AC  students  had  some  level 
of  participation  in  only  35  percent  of  the  active  DL  courses.  AC  stu¬ 
dents  were  almost  never  the  primary  enrollees;  however,  BNCOC  and 
ASI/SQI  course  modules  often  had  a  mixture  of  both.16 

TADLP  objectives  emphasized  the  need  to  reduce  the  time 
allotted  for  learning.  Since  its  beginning,  TADLP  has  emphasized 
the  reduction  in  course  length  to  potentially  yield  savings  in  instruc¬ 
tor,  Temporary  Duty  (TDY),  and  facilities  costs.  For  example,  DL  was 
employed  when  TRADOC  sought  to  reduce  the  length  of  its  BNCOC 
and  ANCOC  courses,  and  when  pilot  courses  were  being  run  to  deter¬ 
mine  whether  CCCs  could  be  converted  from  PCS  to  TDY  courses.  It 
is  noteworthy  that  the  resourcing  formula  for  DL  content  assumes  a  30 
percent  reduction  in  course  hours  over  the  content  that  it  is  replacing. 

Characteristics  of  Emerging  Courseware 

How  does  emerging  courseware  compare  with  active  courseware?  Are 
the  same  characteristics  named  above  present?  We  defined  courses 
to  be  “emerging”  if  they  showed  enrollments,  were  funded,  or  were 
nominated  after  FY  2006.  At  the  time  of  this  writing,  some  previously 
funded  courses  had  become  active  during  FY  2007  and  FY  2008,  after 
our  survey  of  active  courseware.  Further,  new  courses  were  funded  in 


16  While  we  did  not  have  administrative  records  indicating  the  component  of  all  course 
enrollees,  we  did  ask  the  schools  who  they  saw  as  the  most  common  enrollee  for  each  course. 
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FY  2006  and  FY  2007,  and  development  activities  had  begun.17  For  the 
future,  both  TRADOC  and  the  ARNG  had  nominated  courses  to  be 
funded  for  FY  2008  through  FY  2010,  with  only  a  few  decisions  on  FY 
2008  funding  having  been  made  at  the  time  of  this  writing.  In  all,  an 
additional  235  courses,  or  about  40  per  year,  have  been  funded  or  nom¬ 
inated  (and  thus  could  potentially  be  funded)  since  our  initial  analysis. 

We  did  not  have  the  same  detail  available  for  emerging  course¬ 
ware  that  we  did  for  the  active  courses.  However,  we  were  able  to  obtain 
some  administrative  data  from  ATSC  and  to  conduct  interviews  with 
staff  regarding  FY  2006-2010  high-priority  courseware.  Analyzing  all 
the  courseware  funded  to  date,  we  reached  several  conclusions. 

On  the  whole,  many  characteristics  of  emerging  DL  courseware 
have  not  changed  after  2006,  especially  with  respect  to  the  nature 
of  the  courseware  and  the  DL  models  used.  The  dominant  model  of 
recent  and  nominated  courses  has  remained  asynchronous,  self-paced 
IMI,  used  as  part  of  a  phased  learning  model.  The  median  module 
length  is  still  about  60  hours,  with  little  or  no  student-instructor  con¬ 
tact  planned  in  the  DL  phase.  Modules  have  continued  to  aim  pri¬ 
marily  at  knowledge  and  comprehension  learning  levels.  Expected  stu¬ 
dent  load  has  continued  to  vary  considerably,  but  for  most  courses  it 
has  remained  relatively  low,  with  the  median  about  250  students  per 
course. 

Some  trends  began  before  FY  2006  but  are  noteworthy  for  their 
continuation  after  FY  2006.  For  example,  the  percentage  of  high- 
priority  courses  funded  under  TADLP  has  gone  down  considerably 
from  the  earlier  years,  even  though  there  is  still  considerable  unmet 
need  for  shortening  of  the  residential  portion  of  those  courses.  While 
nearly  all  TADLP  courses  funded  before  FY  2003  were  high  priority, 
the  percentage  subsequently  dropped  to  about  50  percent  of  the  funded 
courses  in  FY  2004  and  has  remained  at  that  level  since.  This  trend 
indicates  that  in  the  later  years  of  TADLP,  schools  have  been  more  apt 
to  target  DL  for  other  than  highest-priority  courses.  To  cite  another 


17  DL  courses  are  generally  referred  to  according  to  their  year  of  funding.  Because  courses 
take  several  years  to  develop,  courses  funded  in  FY  2006  and  after  are  not  yet  or  only  just 
beginning  to  become  active  (i.e.,  open  to  enrollments)  at  the  end  of  FY  2008. 
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example,  we  found  that  starting  in  FY  2005,  a  significant  proportion  of 
TADLP-funded  modules  represented  maintenance  or  redesign  of  pre¬ 
viously  funded  courses  rather  than  new  efforts.  From  FY  2005-2007, 
40  percent  of  all  funded  courses  focused  on  maintaining  or  redoing 
prior  DL  content  rather  than  developing  new  DL  content.  The  trend 
appears  to  continue  in  FY  2008. 

The  biggest  difference  between  courseware  funded  pre-  and  post- 
FY  2006  was  an  increase  in  the  percentage  of  funded  ANCOCs  and 
BNCOCs  in  FY  2006,  reflecting  an  increased  emphasis  on  Profes¬ 
sional  Military  Education.  Moreover,  emerging  NCOES  (and  some 
functional)  DL  courses  now  appear  aimed  at  NCOs  in  the  AC,  not 
just  those  in  the  RC.  This  marks  a  shift  in  emphasis  of  DL  to  include 
more  AC  participation. 

Some  evidence  points  to  the  emergence  of  other  types  of  changes 
in  TADLP  courseware.  For  example,  some  Level  4  IMI  courses,  which 
typically  consist  of  a  simulation  or  some  gaming  exercises,  are  being 
funded;  in  FY  2006  and  2007,  Level  4  IMI  accounted  for  5  percent  of 
the  DL  hours  funded.18  Additional  forms  of  student  support  are  also 
emerging;  for  example,  the  Armor  School,  after  completing  a  pilot, 
requested  and  received  authorization  for  RC  instructor  support  for  its 
RC  DL  courses.  Under  this  authority,  RC  instructors  can  be  assigned 
to  an  RC  school  to  actively  monitor  the  progress  of  students  taking  DL 
phases. 

In  addition,  we  see  some  experimentation  with  blended  learn¬ 
ing  models,  in  which  collaborative  DL19  technologies  play  a  signifi¬ 
cant  role  in  the  delivery,  and  instructors  play  a  greatly  increased  role 
in  the  training.  For  example,  AMEDD  has  proposed  its  own  blended 
learning  model  to  be  used  in  residential  training.  Outside  TADLP, 
TRADOC  is  piloting  what  amounts  to  a  new  distributed  blended 
learning  model  in  its  BNCOC  Common  Core  course,  modeled  after 


18  In  general,  IMI  levels  refer  to  the  degree  the  student  must  interact  with  the  IMI  material, 
with  IMI  Level  1  being  the  lowest  and  IMI  Level  4  the  highest.  For  a  complete  definition  of 
IMI  levels,  see  the  next  chapter. 

19  “Collaborative  DL”  is  an  umbrella  term  for  a  variety  of  approaches  to  DL  that  involve 
joint  intellectual  effort  by  students  or  students  and  instructors. 
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an  ANCOC  course  developed  by  Special  Forces.  Salient  features  of  this 
new  approach  to  DL  compared  to  the  traditional  Army  model  are  that 
(1)  a  significant  portion  of  the  course  uses  asynchronous  collaborative 
DL  methods  (2)  the  IMI  portion  of  the  course  has  increased  instruc¬ 
tor  support  in  delivery,  and  (3)  the  IMI  portion  is  developed  by  DL 
instructors  rather  than  by  contractors.  These  new  efforts  will  be  further 
discussed  later  in  this  report. 


Army  Learning  Management  System  (ALMS) 

We  now  describe  the  ALMS,  which  TRADOC  is  implementing  to 
manage  DL  and  support  resident  instruction.  The  goal  of  the  ALMS  is  to 
standardize  and  centralize  learning  management  efforts  in  TRADOC 
schools  and  thus  better  support  worldwide  access  to,  and  tracking  of, 
training  activities,  materials,  and  resources.  Such  management  capa¬ 
bilities  have  been  increasingly  cited  as  a  key  need  for  medium-to-large 
organizations  with  multiple  training  and  professional  development 
requirements  that  are  dispersed  around  the  world  (Schank,  2002).  The 
potential  advantages  of  a  centralized  LMS  include,  but  are  not  limited 
to: 

•  Cost  savings  from  reductions  in  license  fees  and  in  development/ 
maintenance/support  costs  for  software. 

•  Ease  of  use  for  employees  from  single  user  interface,  consistency 
of  data,  and  access  across  all  parts  of  the  organization. 

•  Improved  data  visibility  and  accountability  to  the  central  training 
organization  to  evaluate  course  usage  and  return  on  investment 
from  training  expenditures. 

Development  of  ALMS 

TRADOC’s  centralization  of  learning  management  capabilities  is 
being  carried  out  via  the  design,  development,  and  deployment  of  the 
ALMS.  As  explained  earlier,  the  need  for  the  ALMS  was  identified  by 
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the  TCM  in  his  role  as  combat  developer,  and  the  solution  was  devel¬ 
oped  by  PM  DLS  in  its  role  as  materiel  developer.20 

The  ALMS  has  been  designed  not  as  a  single  system,  but  as  a 
suite  of  integrated  tools  to  meet  a  set  of  required  capabilities.  The 
system  requirements  for  ALMS  development,  through  “Increment  3” 
of  the  DLS  incremental  acquisition  approach,  include  the  capabilities 
(called  “Learning  Lifecycle  Features”)  shown  in  Table  2. 1.21  As  the 
table  indicates,  the  breadth  and  depth  of  these  specified  capabilities  are 
greater  than  most  corporate  demands  for  learning  management  sup¬ 
port,  and  thus  require  more  integrated  features  than  commercial  LMS 
applications. 

The  ALMS  is  expected  to  serve  a  broader  role  than  that  of  a  com¬ 
mercial  or  corporate  LMS,  which  traditionally  provides  the  support  for 
administering  and  tracking  both  DL  and  classroom-based  instruction, 
as  well  as  providing  access  to  and  tracking  use  of  IMI  instructional 
materials.22  The  Army’s  definition  of  capabilities  also  includes  aspects 
of  a  “learning  content  management  system”  (e.g.,  “product  catalog”  of 
learning  objects),  as  well  as  collaboration  tools  for  DL  delivery. 

The  Army  materiel  developer,  PM  DLS,  selected  commercial  off- 
the-shelf  (COTS)  solutions  to  meet  the  requirements  in  Table  2.1,  and 


20  The  Operational  Requirements  Document  (ORD)  for  the  ALMS  was  published  in  1999 
and  revised  in  2002.  The  ALMS  “Engineering  and  Manufacturing  Development  (EMD)” 
began  August  2000,  with  the  COTS  vendor-selection  process  taking  place  in  2001.  Produc¬ 
tion  of  the  ALMS  began  in  September  2004. 

21  Note  that  while  career  tracking  or  management  is  sometimes  discussed  in  the  context  of 
the  ALMS,  at  the  time  these  requirements  were  specified  they  did  not  call  for  that  capability. 

22  Although  the  technical  definitions  of  learning  management  systems  vary  widely,  Rossett 
(2002),  the  American  Society  for  Training  and  Development’s  E-Learning  Handbook,  defines 
a  learning  management  system  as  “Software  that  automates  the  administration  of  training 
events.  The  LMS  registers  users,  tracks  courses  in  a  catalog,  and  records  data  from  learners; 
it  also  provides  appropriate  reports  to  management.  The  database  capabilities  of  the  LMS 
extend  to  additional  functions  such  as  company  management,  online  assessments,  person¬ 
alization,  and  other  resources.”  The  handbook  goes  on  to  state  that  “Learning  management 
systems  administer  and  track  both  online  and  classroom-based  learning  events,  as  well  as 
other  training  processes  (these  would  need  to  be  manually  entered  into  the  system  for  track¬ 
ing  purposes).  An  LMS  is  typically  designed  for  multiple  publishers  and  providers.  It  usually 
does  not  include  its  own  authoring  capabilities;  instead  it  focuses  on  managing  courses  cre¬ 
ated  from  a  variety  of  other  sources.” 
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Table  2.1 

Learning  Lifecycle  Features  of  ALMS 


Course  Catalog:  A  "unified,  on-line  central  directory  for  Department  of  Army 
(DA)  course  offerings.  Learners  will  be  able  to  locate  and  register  for  courses  and 
initiate  e-courses." 

Product  Catalog:  An  "on-line  central  directory"  allowing  learners  and  course 
developers  to  order  two  types  of  products: 

•  Learning  support  products  for  learners. 

•  Learning  objects  for  course  developers  (e.g.,  items  created,  reused,  and 
combined  to  create  courses,  phases,  modules,  lessons,  activities,  or  media). 

Registration:  A  capability  to  make  a  "reservation  to  attend  a  training  event  (e.g., 
resident,  nonresident)." 

Resource  and  Event  Scheduling:  Allows  "planning  training  events  (e.g., 
synchronous)  and  arranging  for  resources  (e.g.,  supplies,  facilities,  instructors)  to 
support  the  event."  A  training  event  may  include: 

•  Resident  education/training. 

•  Nonresident  education/training. 

•  Special  events. 

Product  Distribution:  Involves  "the  distribution  of  courseware  and  products  to 
Learners,  Instructors  and  Class  Managers." 

Training  Delivery:  Supports  the  "full  cycle  of  the  training  event.  The  cycle  begins 
when  the  Learner  arrives  for  resident  training  or  receives  nonresident  training 
materials  and  ends  with  the  completion  of  instruction  and  the  update  of  the 
Learner's  Individual  Training  Record  (ITR)." 

Testing:  Includes  "measuring  knowledge,  learning,  and  proficiency  in  order 
to  meet  defined  learning  objectives.  Testing  involves  the  compilation  and 
instantiation,  administration,  scoring,  and  recording  of  a  Learner's  assessed 
performance  competence." 

Collaboration:  "Tools  enhance  the  learning  experience  by  enabling  Learners  and 
Instructors  to  bridge  distances  in  geography  and  time.  They  provide  a  means 
of  virtually  engaging  in  learning  events  and  knowledge  sharing  through  a 
synchronous,  real-time  exchange  or  an  asynchronous,  non  real-time  exchange." 

Evaluation:  The  collection  of  course-related  feedback  from  the  Learner  during 
or  after  the  completion  of  a  learning  event.  Evaluation  involves  the  creation, 
administration  and  recording  of  the  Learner's  experience  with  instructional 
materials,  Instructors  and  the  application  of  newly  acquired  skills." 

Training  Management:  "Defines  future,  standardized  requirements  of  DLS  to 
collect,  process,  and  output  information  on  the  testing  of  courseware,  maintaining 
the  currency  of  courseware  information,  and  supporting  asset  management  to 
ensure  the  successful  execution  of  training  events." 

Course  Scheduling:  "Defines  a  specific  instance  of  a  course  (i.e.,  course  iteration) 
and  includes  the  date,  times  and  location  of  the  event.  Course  scheduling  applies 
to  resident  and  nonresident  training  events." 
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has  carried  out  acquisition  and  integration  of  applications  up  through 
ALMS,  Version  2.0.  These  COTS  applications  and  suites  of  applica¬ 
tions  provide  a  variety  of  capabilities  to  satisfy  the  requirements  of  the 
DLS  Increment  3  ALMS  specifications: 

•  Saba  Enterprise  5.3  Learning  Suite: 

-  Origins  in  commercial  e-learning,  broadened  market  to  include 
academic  use. 

•  Blackboard  Academic  Suite,  Enterprise: 

-  Origins  in  academic  institutional  use,  now  a  leading  LMS  for 
both  academic  and  commercial  training  organizations. 

•  Saba  “Centra”  for  collaboration: 

-  Application  provides  “virtual  classroom”  learning  (synchro¬ 
nous  or  asynchronous  if  recorded  synchronous  session  is  later 
viewed  as  a  movie),  e-meeting,  and  Web  seminar  platform 
with  a  learning  content  management  capability.  Also  capable 
of  including  video  for  collaboration,  or  a  version  of  “desktop 
video  teleconference.” 

Any  integration  effort  for  a  suite  of  COTS  solutions  from  differ¬ 
ent  providers  results  in  overlapping  features  offered  by  the  competing 
applications.  Given  that  each  application  is  implemented  in  a  particu¬ 
lar  way,  the  overlap  in  solutions  allows  organizations  to  choose  which 
application  will  most  effectively  meet  their  local  LMS  needs. 

Status  of  ALMS  Use 

Considering  the  low  impact  of  high-priority  courseware  in  training 
thus  far,  we  did  not  attempt  to  gather  specific  information  about  how 
many  of  those  courses  were  supported  out  of  the  ALMS.  However, 
from  interviews,  we  determined  the  overall  status  of  data  availability 
for  this  part  of  the  DL  program. 

While  several  hundred  courses  were  loaded  into  the  ALMS  at  the 
end  of  FY  2007,  few  were  of  the  high-priority  type  that  is  the  focus 
of  this  study.  Migration  of  DL  courses  supported  by  other  LMSs  to 
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the  ALMS  is  an  objective  of  a  number  of  initiatives.  Once  courses  are 
loaded  onto  the  ALMS,  usage  data  will  be  more  readily  available. 

Future  Directions  of  ALMS 

The  DLS  Increment  3,  “Automated  Student  Management,”  was  sched¬ 
uled  to  move  into  the  final  implementation  task  of  “Post-Deployment 
System  Support”  on  September  27,  2004.  The  system  was  completely 
fielded  but  not  widely  in  use.  Full  implementation  requires  comple¬ 
tion  of  transition  activities  (e.g.,  migration  of  courses  from  the  existing 
LMS  to  the  ALMS)  at  each  school. 

The  PM  DLS  has  developed  and  implemented  a  structured 
“change  management  process”  for  ALMS  software  revisions.  This  is  the 
process  by  which  PM  DLS  collects  feedback  from  users  and  proponent 
schools  via  a  “change  request,”  works  with  the  TCM  to  evaluate  and 
prioritize  possible  changes,  and  determines  the  costs  of  these  changes. 
Such  changes  can  include  (from  the  ALMS  Engineering  Change  Pro¬ 
posal  website23): 

•  User-viewpoint  observations  of  undesirable  system  behavior. 

•  Suggestions  for  improvements  on  existing  processes. 

•  Ideas  for  new  functions  or  processes. 

•  Requests  for  information  about  a  particular  feature. 

•  Problems  in  following  user  documentation  (Help  files). 

Once  changes  are  approved  by  the  TCM  and  funded,  a  requested 
change  is  moved  through  the  software  development  process  to  imple¬ 
ment  the  change. 

The  DLS  has  moved  on  to  Increment  4  development  (the  final 
increment),  which  does  not  appear  to  provide  any  additional  capa¬ 
bilities  to  the  ALMS  features  specified  above.  Instead,  this  increment 
focuses  on  the  development  of  “Deployed  Digital  Training  Campuses” 
(DDTCs),  described  in  the  next  section. 


23  https://www.dls.army.mil/CR_Process.html 
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DL  Facilities 

The  Army  DL  program  also  supports  a  worldwide  network  of  facilities 
that  allow  access  to  distributed  learning.  As  of  the  end  of  FY  2007, 
these  include  the  following: 

•  Digital  Training  Facilities  (DTFs):  About  230  DTFs  exist  on 
a  few  more  than  90  CONUS  (continental  United  States)  and 
OCONUS  (outside  the  continental  United  States)  installations. 
The  facilities  support  DL  access  (about  3,500  workstations),  video 
teletraining  receipt,  and  video  conferencing. 

•  ARNG  DL  Classrooms:  About  340  ARNG  DL  classrooms 
throughout  the  54  states  and  territories,  many  of  which  sup¬ 
port  interactive  courses  involving  face-to-face  student-instructor 
contact. 

•  Classroom  XXI:  About  84  classrooms  on  TRADOC  school 
installations  supporting  resident  training  and  with  the  capability 
to  receive  and  transmit  DL  courses. 

•  Deployed  Digital  Training  Campuses:  Eventually,  around  50 
mobile,  networked  systems  to  train  soldiers  who  are  outside 
CONUS  and  who  cannot  gain  access  to  other  facilities.  Two  are 
planned  for  each  Army  division,  with  others  held  in  reserve  for 
surge  support.  The  systems  consist  of  about  20  notebook  comput¬ 
ers,  a  server  and  printer,  and  equipment  to  transfer  digital  infor¬ 
mation  to  and  from  satellites. 

These  facilities  typically  serve  as  more  than  a  location  where  sol¬ 
diers  can  take  TADLP  courses.  For  example,  DTFs  provide  “support 
of  ongoing  operations”  and  “required  mission  immediate  training  for 
mobilization,  activation,  and  deployment.”  Part  of  the  purpose  of  the 
ARNG  DL  classrooms  is  to  enhance  mobilization  capabilities  and 
facilitate  command  and  control.  The  Classroom  XXIs  generally  sup¬ 
port  residential  training  at  TRADOC  schools.  Further,  new  uses  for 
DL  facilities  emerge  regularly. 

Available  data  on  facilities  focus  on  calculating  the  usage  of  facil¬ 
ities  to  ensure  that  there  is  enough  justification  for  their  continued 


Features  of  TADLP  29 


funding.  However,  at  the  time  of  this  study,  systematic  data  were  not 
collected  with  regard  to  the  specific  purposes  customers  have  for  the 
facilities  they  use.  While  logs  would  allow  the  compilation  of  some  gen¬ 
eral  categories  of  usage,  it  was  not  possible  to  calculate  all  the  instances 
in  which  facilities  were  used  for  structured  institutional  courses,  such 
as  reclassification  courses,  ANCOCs,  or  BNCOCs. 


IMI  Development  Process  Description 

In  this  section  we  review  the  process  for  IMI  production  used  by 
TADLP  at  the  end  of  FY  2007.  This  description  is  included  as  a  “fea¬ 
ture”  of  TADLP  because  it  is  important  background  for  a  number  of 
our  recommendations.  The  Army’s  IMI  development  process  is  docu¬ 
mented  in  TRADOC  Pamphlets  350-70-12,  Distributed  Learning — 
Managing  Courseware  Production  and  Implementation ,  and  350-70-2, 
Multimedia  Courseware  Development  Guide.  These  publications  pro¬ 
vide  detailed  guidance  on  the  following  phases  of  taking  an  IMI  course 
from  a  “need”  to  an  implemented  course  being  used  by  soldiers: 

•  Courseware  production  pre-award  phase. 

•  Courseware  production  development  phase. 

•  Courseware  production  fielding  phase. 

This  guidance  is  aligned  with  the  Army’s  broader  “Systems  Approach 
to  Training”  (SAT)  training  development  process  documented  in 
TRADOC  Regulation  350-70,  Systems  Approach  to  Training  Manage¬ 
ment,  Processes,  and  Products.24  The  general  process  is  described  as  “a  sys¬ 
tematic,  iterative,  spiral  approach  to  making  collective,  individual,  and 
self-development  education/training  decisions  for  the  Army.”  It  is  this 
process  that  ATSC  follows  in  the  production  of  TADLP  courseware. 

To  facilitate  better  understanding  of  the  Army’s  process  for  devel¬ 
oping  IMI,  our  project  sought  to  identify  the  key  steps  in  the  “end-to- 


24  TRADOC  Regulation  350-70  and  the  supporting  pamphlets  are  currently  undergoing 
significant  revision. 
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end”  IMI  development  process.  This  process  reflects  the  perspectives  of 
each  stakeholder  in  the  process,  e.g.,  the  proponent  school  with  the  need 
for  the  course,  TRADOC’s  need  for  financial  authority  and  account¬ 
ability,  and  course  developers’  need  for  information  and  collaboration. 

To  complete  our  “process  mapping,”  we  took  into  account  the 
guidance  in  TRADOC  Pamphlet  350-70-12  (referenced  above),  as 
well  as  information  and  data  from  interviews  with  Army  civilians  at 
ATSC,  and  interviews  with  training  developers  and  instructors  at  the  a 
number  of  the  Army’s  proponent  schools. 

An  important  part  of  any  process  mapping  is  finding  discrete 
points  in  that  process  where  important  events  can  be  identified  and 
time-stamped.  We  divided  the  Army’s  IMI  development  process  into 
two  major  segments.  The  first  segment  encompasses  all  the  activities 
from  when  the  school  first  has  a  perceived  need  for  an  IMI  course 
through  a  signed  contract  for  the  work  to  develop  the  course.  While 
early  on  some  DL  courses  were  developed  within  the  schoolhouse, 
virtually  all  DL  content  funded  under  TADLP  is  now  produced  by 
TRADOC-approved  contractors  with  the  support  of  the  schools.  The 
second  segment  covers  all  activities  from  the  signing  of  the  contract 
(or  delivery  order)  until  the  time  when  there  are  actual  Army  learners 
using  the  IMI  content. 

The  top  row  of  Figure  2.1  below  shows  the  main  activities  in  the 
first  major  segment:  from  perceived  need  for  the  course  to  signed  deliv¬ 
ery  order.  This  segment  is  also  referred  to  as  the  “courseware  produc¬ 
tion  pre-award  phase”  of  the  DL  development.  Although  the  processes 
shown  in  Figure  2.1  appear  to  be  simple  in  this  representation,  the 
actual  underlying  processes  are  complex,  with  many  steps  and  many 
supporting  documents  and  stakeholders. 

The  first  step  is  for  the  schools  to  identify  a  need  within  a  course 
for  an  appropriate  piece  of  the  curriculum  to  be  developed  using 
IMI.  TRADOC  Regulation  350-70,  II-9-6,  contains  general  “media 
selection  guidelines”  with  factors  for  consideration  of  “training  effec¬ 
tiveness,”  “DL  facility  location  and  capabilities,”  “affordability,”  and 
“budget  limitations.” 

Once  the  school  has  assessed  a  need  for  course  content  that  can 
be  filled  appropriately  by  a  piece  of  IMI  courseware,  the  school  must 
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Figure  2.1 

Army  DL  Development  Process  Lifecycle 


RAND  MG1016-2. 1 


officially  “nominate”  that  content  for  development  into  IMI  products 
via  TRADOC’s  TADLP  prioritization  process.  Once  nominated,  a 
potential  IMI  course  component  becomes  part  of  a  list  at  TRADOC 
that  is  reviewed,  that  ranks  the  courses  according  to  their  importance, 
and  that  will  be  used  to  determine  which  courses  will  be  funded. 
Funding  decisions  are  based  on  a  set  of  criteria  for  course  necessities 
that  includes,  among  other  criteria,  the  estimated  number  of  users  over 
the  lifecycle  of  the  course,  the  importance  of  the  course  content,  and 
costs.  Since  this  selection  process  takes  place  in  the  February-March 
time  frame  and  occurs  only  once  per  year,  there  is  an  average  wait  of 
six  months  for  a  course  to  be  selected  from  the  time  when  the  school 
identifies  a  need  for  the  course. 

Once  course  prioritization  decisions  are  made,  the  schools  must 
start  the  process  of  producing  a  delivery  order  (DO);  develop  and 
submit  course  administrative  data  (CAD);  provide  government  fur¬ 
nished  information  (GFI)  and  government  furnished  material  (GFM) 
that  provides  full  documentation  of  the  course,  including  goals,  train¬ 
ing  support  packages,  methods,  and  assessments;  and  also  submit 
responses  to  a  number  of  supplemental  questions.  These  documents 
then  flow  through  the  various  contracting  offices  for  review,  supple¬ 
menting  as  needed  using  various  forms,  and  approvals.  Eventually 
a  contracting  officer  (KO)  approves  a  purchase  order  for  the  course 
development,  and  the  contracting  office  sends  requests  for  proposals 
(RFPs)  to  a  list  of  approved  contractors.  These  contractors  then  submit 
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their  proposals  for  course  development.  These  proposals  are  based  on 
previously  negotiated  rates  of  dollars  per  hour  for  content  development 
for  different  levels  of  IMI,  and  they  are  independent  of  the  content 
area  in  any  specific  RFP.  The  KO  then  has  a  technical  review  board 
review  the  proposals  from  the  contractors,  and  finally  the  KO  awards 
the  contracts.  These  contracts  are  classified  as  “fixed  price,”  meaning 
the  requirements  are  clearly  specified  in  advance,  with  the  contractor 
expected  to  provide  the  IMI  product  at  the  cost  fixed  in  the  contract. 
The  final  awards  and  contract  signatures  are  traditionally  made  in  the 
closing  weeks  of  the  fiscal  year.  This  completes  the  first  major  step  of 
the  IMI  development  process. 

The  next  major  segment  of  the  DL  development  process  takes  us 
from  contract  signature  and  work  commencement  to  the  time  when 
Army  learners  are  actually  using  the  courseware.  This  segment  of  the 
process,  the  second  line  of  boxes  in  Figure  2.1,  involves  fewer  docu¬ 
ments  and  stakeholders  and  is  less  complex  administratively,  from  the 
perspective  of  the  Army. 

After  the  contract  is  awarded,  the  “development  phase”  of  the 
process  begins.  This  phase  starts  with  a  post-award  “kickoff”  meeting 
between  the  contractor  and  representatives  from  both  the  proponent 
school  and  ATSC.  At  this  meeting,  the  contractors  get  their  first  views 
of  the  quality/quantity  of  GFI  provided  for  the  course.  The  contractors 
are  required  to  identify  subject  matter  experts  (SMEs)  to  support  the 
course  development  and  submit  resumes  for  review  by  the  schools. 

The  contractor  then  prepares  a  number  of  courseware  develop¬ 
ment  planning  and  execution  documents  based  on  the  GFI,  the  kickoff 
meeting,  and  subsequent  communications  with  proponent  school  and 
ATSC  representatives.  These  include  a  milestone  schedule,  a  validation 
plan,  a  test  and  evaluation  plan,  a  student  evaluation  plan,  an  instruc¬ 
tional  media  design  package  (IMDP),  and  a  prototype  lesson  that 
demonstrates  understanding  of  the  educational  and  technical  require¬ 
ments  of  the  courseware.  Once  these  documents  are  reviewed  by  the 
proponent  schools  and  ATSC  courseware  manager  and  subsequently 
approved,  courseware  development  begins. 

Contractor  development  processes  vary  by  contractor  and  pro¬ 
ponent  school.  Tessons  are  developed  by  the  contractor  for  the  school 


Features  of  TADLP  33 


either  individually  or  in  small  numbers  and  submitted  for  approval  by 
the  proponent  school  (with  a  10-day  suspense  time).  If  approved,  the 
lessons  are  submitted  for  testing  by  the  Courseware  Certification  Team 
(CCT)  at  ATSC  for  basic  technical  testing.  The  testing  at  this  point 
does  not  include  Sharable  Content  Object  Reference  Model  (SCORM) 
conformance25  or  ALMS  certification.  As  a  lesson,  or  sometimes  a  small 
number  of  lessons,  is  approved  and  passes  testing,  payment  is  made  to 
the  contractor  for  those  completed  lessons. 

When  all  lessons  have  been  approved  and  passed  initial  testing, 
the  entire  set  of  lessons  is  submitted  as  delivery  of  a  completed  final 
course.  The  full  course  is  then  submitted  to  CCT  for  full  SCORM 
testing  and  ALMS  compliance.  When  the  cycle  of  testing,  reporting 
issues,  contractors  addressing  issues,  and  retesting  has  been  completed, 
the  course  is  uploaded  to  the  ALMS  and  is  available  for  access  by  Army 
learners.  At  this  time  also  the  final  contract  payments  are  made  and 
the  contract  closed.  The  courses  have  a  six-month  warranty  period  for 
reporting  defects  after  the  contract  has  been  closed.  There  is  then  some 
period  of  time  between  when  the  course  is  “completed”  from  a  contrac¬ 
tual  standpoint,  and  when  that  course  is  used  by  Army  learners.  This  is 
sometimes  due  to  the  demand  cycle  for  the  IMI  component  of  a  course: 
some  courses  are  offered  only  once  a  year  by  a  school,  so  the  newly 
delivered  IMI  is  not  used  until  that  phase  of  the  next  course  offering 
comes  up  on  the  school’s  training  schedule. 

Summing  up  the  second  major  segment  of  the  IMI  development 
process,  we  can  say  that  the  process  has  fewer  steps  and  stakeholders 
than  the  first  segment,  but  requires  more  interaction  by  contractors  and 
proponent  schools.  Moreover,  there  are  important  testing  and  quality 
assurance  steps  by  TRADOC  DL  organizations  late  in  the  process  as 
the  IMI  materials  are  certified  to  conform  to  standards  and  to  load  and 
function  appropriately  with  learning  management  systems. 


25  SCORM  is  a  collection  of  standards  and  specifications  for  web-based  e-learning. 
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Chapter  Summary 

TADLP,  in  sum,  is  primarily  composed  of  IMI  courseware,  an  ALMS, 
and  facilities  that  can  support  learners  taking  DL  courses.  In  this  chap¬ 
ter  we  have  described  the  funding,  features,  and  characteristics  of  each 
of  those  components  in  the  FY  2006-2008  time  frame.  Our  emphasis 
has  been  on  courseware,  which  we  argue  is  the  program’s  key  compo¬ 
nent  and  which  we  thus  make  the  primary  focus  of  this  report.  This 
description  will  serve  as  context  and  background  for  our  assessment  of 
the  current  state  of  DL,  and  for  our  recommendations  with  regard  to 
improvements. 


CHAPTER  THREE 


The  State  of  Army  DL 


Now  that  we  have  established  the  context  for  TADLP,  we  describe  the 
state  of  DL  as  of  FY  2006,  addressing  the  first  purpose  of  the  project 
and  the  first  stage  of  our  analysis  as  presented  in  Figure  I.2.1  In  keep¬ 
ing  with  our  focus  on  courseware,  we  report  results  for  the  five  mea¬ 
sures  of  effectiveness  we  developed.  The  first  three  areas  listed  below 
concern  DL  outcomes,  while  the  remaining  two  concern  the  process  of 
developing  courseware: 

•  Program  impact:  What  has  been  the  impact  of  TADLP  course¬ 
ware  in  terms  of  number  of  courses  developed,  students  served, 
and  training  hours  conducted?  To  what  extent  has  TADLP 
courseware  become  a  part  of  Army  training? 

•  Efficiency:  How  efficiently  have  TADLP’s  course  development 
resources  been  utilized,  based  on  the  impact  of  courseware  and 
other  intermediary  outputs  relative  to  costs? 

•  Quality:  What  has  been  the  quality  of  courseware  output? 

•  Cycle  time:  How  much  time  has  been  required  for  courseware 
production? 

•  Responsiveness:  How  readily  can  the  process  for  producing 
courseware  adapt  in  response  to  changing  requirements? 

Information  about  all  these  measures  is  important  to  understand¬ 
ing  the  state  of  TADLP.  Findings  from  these  measures  will  provide  the 


1  We  also  determined  that  the  trends  we  saw  in  the  FY  2006  data  did  not  change  markedly 
through  FY  2008,  the  end  of  our  assessment  period. 
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basis  for  further  analyses  and  recommended  improvements  to  the  IMI 
program,  which  will  be  presented  in  Chapter  Four.  Given  that  a  system 
of  overarching  metrics  did  not  exist  prior  to  our  assessment,  the  data 
presented  below  can  also  be  used  to  establish  a  baseline  of  performance 
for  tracking  future  improvements. 

Program  Impact 

We  first  consider  the  impact  of  the  DL  program  on  Army  training. 
We  looked  at  the  number  of  courses  provided,  the  students  served  (AC 
and/or  RC),  and  the  number  of  training  hours  conducted  compared  to 
those  used  for  residential  training. 

Number  of  Active  DL  Courses  Is  Small 

Our  survey  of  DL  courses  provided  information  about  the  number 
and  type  of  “priority  one”  DL  courses  active  in  FY  2006.  We  focused 
on  priority  one  courses  because  of  the  key  role  these  courses  play  in 
the  Army  Training  and  Leader  Development  Strategy  (ATLDS)  and 
because  they  have  represented  an  Army  focus  in  its  DL  program  to 
date.  This  survey  relied  not  only  on  records  from  the  Army  Train¬ 
ing  Requirements  and  Resources  System  (ATRRS)  but  also  on  queries 
to  proponent  schools  about  courses  or  enrollments  that  may  not  have 
been  listed  on  ATRRS.  Key  findings  are  represented  in  Figure  3.1  and 
summarized  below. 

During  FY  2006,  103  DL  courses  or  modules  within  priority 
one  categories  were  in  use  (i.e.,  had  enrollments).2  Of  these,  about  92 
appear3  to  have  been  funded  through  TADLP  and  the  active  compo¬ 
nent.4  As  shown  in  Figure  3.1,  reclassification  courses  were  the  most 


2  Active  correspondence  courses  were  excluded  from  this  count,  unless  they  were  a  course 
requirement.  Functional  DL  courses  were  also  excluded,  unless  they  had  an  ASI  or  SQI 
designation. 

3  Since  there  is  no  unique  identifier  given  to  a  DL  course  funded  through  TADLP,  it  is 
sometimes  difficult  to  determine  whether  a  course  in  ATRRS  was  created  through  the  pro¬ 
gram  or  not. 

4  Most  of  the  remainder  were  funded  by  the  ARNG,  which  in  recent  years  has  obtained 
independent  funding  for  the  production  of  digital  training  content. 


The  State  of  Army  DL  37 


Figure  3.1 

Number  and  Type  of  DL  Priority  One  Courses  in  FY  2006 


Number  of  courses 
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common  subject  matter  for  DL  courseware,  and  ANCOC  and  ASI/ 
SQI  courses  the  least  common. 

Limited  Use  of  DL  for  AC 

Most  DL  instruction  was  taken  by  RC  soldiers.  Figure  3.1  also  indi¬ 
cates  that  RC  soldiers  and  leaders  were  the  most  common  targeted 
audience  for  DL  courses,  especially  in  reclassification  and  Captains 
Career  courses.5  AC  students  were  almost  never  the  primary  enrollees 
of  a  DL  course.  However,  DL  courses  in  the  BNCOC  and  ASI/SQI 
categories  often  had  a  mixture  of  both  AC  and  RC  students.  Overall, 
schools  reported  that  AC  students  had  some  level  of  participation  in  35 
percent  of  the  active  DL  courses. 


5  Although  we  did  not  have  administrative  records  indicating  the  component  of  course 
enrollees,  we  did  ask  the  schools  who  they  saw  as  the  most  common  enrollee  for  each  course. 
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DL  Amounts  to  a  Small  Percentage  of  Training 

DL  courses  accounted  for  only  a  small  percentage  of  structured  train¬ 
ing  courses  offered  by  the  Army.  Across  the  five  types  of  courses  con¬ 
sidered,  DL  training  accounted  for  less  than  6  percent  of  all  train¬ 
ing  hours6  across  all  course  categories  except  ASI/SQI  (see  Figure  3.2). 
Courses  in  this  category,  particularly  the  Battle  Staff  Course  and  the 
First  Sergeant  Course,  represented  some  of  the  biggest  success  stories 
for  DL.7  However,  DL  was  seldom  used  in  the  other  four  categories  of 
courses.  In  ANCOC,  for  example,  DL  accounted  for  only  about  6  per¬ 
cent  of  all  the  training.  The  lowest  percentage  of  DL  courseware  was 
for  the  reclassification  category,  which  accounted  for  only  2  percent  of 
total  training.  This  result  is  somewhat  surprising,  since  reclassification 
for  the  RC  was  intended  to  be  a  focus  of  TADLP  from  its  inception. 

While  the  percentage  of  DL  courses  in  total  training  is  clearly  less 
than  desired,  it  is  not  clear  what  an  optimum  percentage  would  be.  We 
found  some  targets  in  TADLP  Campaign  Plan,  a  2001  publication  that 
provides  an  example  of  the  vision  the  Army  has  for  DL  eventually.  The 
report  stated  that  one  “critical”  indicator  of  success  would  be  for  DL  to 
satisfy  “between  30-65  percent  of  the  quota-managed  training  load” 
(see  Section  1.9.4  of  the  plan).  Figure  3.2  displays  that  goal  relative  to 
what  has  been  achieved. 


Efficiency  of  Utilization  of  Courseware  Resources 

We  now  consider  the  efficiency  of  DL  courseware  production.  Effi¬ 
ciency  in  courseware  production  can  be  viewed  in  terms  of  the  cost  of 
content  development  per  hour  of  training  executed.  While  readiness  needs 
might,  in  some  instances,  warrant  investing  in  IMI  to  train  only  a  few 
students,  an  efficient  program  would  typically  develop  DL  courses  that 


6  “Training  hours”  in  a  category  were  calculated  by  taking  the  product  of  course  length 
times  the  number  of  FY  2006  graduates  for  each  course,  then  summing  over  all  courses  in 
the  category. 

7  It  should  be  noted  that  these  DL  courses  use  VTT  as  the  main  delivery  mode,  and  VTT 
courseware  is  not  developed  under  TADLP. 
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Figure  3.2 

DL  Training  Hours  as  a  Percentage  of  All  Training  (FY  2006) 


Usage  in  FY06 


SOURCE:  ATRRS  FY06. 
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offer  a  high  return  on  investment  (ROI),  that  is,  content  with  likely 
high  usage  and  relatively  low  cost.  While  we  did  not  attempt  a  formal 
calculation  of  ROI,  we  did  look  at  both  cost  and  usage,  and  we  used  a 
comparative  approach  to  draw  conclusions  about  efficiency.8 

Cost  per  Hour  Is  Relatively  Low 

We  first  looked  at  the  cost  aspect  of  efficiency;  that  is,  the  cost  the 
Army  paid  for  its  development  contracts.  (Later  in  this  chapter  we 
examine  what  the  Army  was  able  to  achieve  as  a  result  of  its  invest¬ 
ment.)  To  attain  some  perspective  with  regard  to  cost,  we  compared 
what  the  Army  paid  to  what  is  typically  paid  in  the  commercial  sector 
for  comparable  content.  In  order  to  compare  content  requiring  about 
the  same  level  of  effort  to  develop,  we  divided  courses  into  two  catego- 


8  At  DA  level,  the  primary  metric  for  evaluating  courseware  within  TADLP  has  histori¬ 
cally  been  the  “number  of  courses  produced.”  However,  a  recent  initiative  to  produce  addi¬ 
tional  metrics  has  called  for  employing  courseware  “usage”  as  a  measure  directly  related  to 
efficiency. 
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ries:  Level  2  or  Level  3  (grouped  together)  and  Level  4,  as  defined  by 
the  Army.  The  Army  defines  four  levels  of  IMI: 

•  Level  1  IMI  signifies  that  the  learner  is  a  passive  recipient  of 
information. 

•  Level  2  IMI  involves  more  learner  interaction  and  control  over 
lesson  activities,  e.g.,  clicking  on  icons  to  reveal  information, 
moving  objects  on  the  screen,  filling  in  forms,  and  answering 
questions. 

•  Level  3  IMI  includes  more  involved  participation,  such  as  the  use 
of  scenarios  for  testing,  the  need  for  the  learner  to  make  decisions, 
and  more  complex  branching  based  on  the  learner’s  responses. 

•  Level  4  IMI  goes  even  further  toward  immersive  participation 
and,  in  particular,  involves  simulations. 

The  industry  survey  did  not  use  exactly  the  same  definitions  for 
levels  of  activity  as  the  Army  did  for  the  levels  below  4.  However,  we 
assigned  a  Level  2-3  to  correspond  to  the  following  industry  defini¬ 
tion:  Involves  sophisticated  navigational  controls  and  mid-level  inter¬ 
action  with  the  content,  with  at  least  20  percent  of  content  devoted  to 
very  interactive  learning  activities  (such  as  discovery  exercises,  drag- 
and-drop  exercises,  etc.).  Figure  3.3  shows  the  result  of  the  comparison 
of  Level  2—3  and  Level  4  IMI. 

The  Army  was  able  to  negotiate  a  favorable  price  for  its  DL  courses. 
For  the  bulk  of  its  content  (Level  2-3),  the  Army  paid  much  less,  on 
average,  than  the  commercial  sector.  The  bars  on  the  left  of  the  figure 
show  that  the  Army  paid  an  average  of  $14,000  per  hour  for  Level  2-3 
interactivity,  whereas  commercial  customers  paid  about  $26,000  per 
hour,  almost  twice  as  much.9  Average  prices  for  Level  4  IMI  were  about 
the  same  for  the  Army  as  in  the  commercial  sector  (see  the  set  of  bars 
on  the  right),  although  the  small  sample  sizes  make  this  comparison 
tentative. 


9  The  commercial  prices,  which  came  from  a  variety  of  vendors  in  a  variety  of  contracts,  also 
had  larger  variances  associated  with  the  average.  In  the  case  of  the  Army,  a  single  customer 
with  relatively  similar  needs  across  delivery  orders,  the  actual  price  varied  from  $10,000  per 
academic  hour  to  about  $22,000. 


The  State  of  Army  DL  41 


Figure  3.3 

Comparison  of  Average  Army  and  Commercial  Costs  for  IMI 
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The  lower  costs  paid  by  the  Army  make  sense  in  light  of  the 
potential  buying  power  of  the  Army  compared  to  the  average  industry 
buyer.  It  appears  that  the  Army  has  been  able  to  negotiate  relatively  low 
prices  due  to  its  size. 

However,  other  factors  must  also  be  considered  in  order  to  deter¬ 
mine  the  overall  efficiency  of  the  Army  program.  First,  this  metric 
needs  to  be  considered  in  the  light  of  the  quality  of  courseware  that  is 
produced,  a  subject  that  we  take  up  later  in  this  chapter.  If  the  output 
does  not  measure  up  to  all  the  Army  standards,  the  low  prices  may  not 
lead  to  any  ultimate  benefit.  Second,  there  is  a  question  of  whether  the 
Army  is  investing  in  content  for  which  usage  is  likely  to  be  high.  If 
only  a  small  number  of  students  are  using  DL  content,  then  even  the 
low  prices  paid  by  the  Army  will  be  relatively  high  on  a  usage  basis.  We 
take  up  the  effect  of  “usage”  on  efficiency  below. 

Most  DL  Courses  Do  Not  Produce  Graduates  or  Are  Withdrawn 
Shortly  After  Fielding 

Administrative  data  collected  by  ATSC  regarding  the  status  of  Army 
IMI  development  efforts  can  provide  additional  insights  into  the  value 
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the  Army  receives  for  its  DL  investments,  particularly  in  terms  of  usage. 
ATSC  uses  a  number  of  “status”  categories  for  its  DL  efforts  which  can 
be  used  to  define  alternative  dispositions  for  courses.  Below  we  define 
the  categories  we  were  able  to  create: 

•  Removed:  Courses  were  abandoned  before  completion,  exclud¬ 
ing  those  that  were  nominated  but  dropped  because  funding 
never  became  available.  In  some  cases  significant  resources  were 
expended  before  abandonment,  but  in  other  cases  expenditures 
may  have  been  minimal. 

•  Outstanding:  These  courses  were  still  in  progress  for  develop¬ 
ment  (as  of  the  first  quarter  of  FY  2008). 

•  Completed  but  not  fielded:  While  ATSC  shows  that  the  course¬ 
ware  was  finished  and  approved,  the  content  for  these  courses  was 
never  fielded  (i.e.,  offered  to  potential  students)  by  the  proponent 
school. 

•  Fielded:  Here  ATSC  records  show  that  the  courseware  was  fielded 
by  the  school. 

Whether  fielded  courses  produced  graduates  or  not  can  be 
determined  from  ATRRS  data  records  for  all  DA-required  courses. 
Because  usage  is  not  always  recorded  in  ATTRS  for  all  DL  courses,  we 
restricted  our  assessment  to  the  priority  one  courses  that  were  the  focus 
of  our  analysis,  that  is,  reclassification  courses,  BNCOC  and  ANCOC 
courses,  Captains  Career  Courses,  and  key  functional  courses.  Of  the 
375  course  modules  funded  under  TADLP  through  the  end  of  FY 
2005,  227,  or  about  60  percent,  were  within  our  priority  one  categories 
of  interest. 

Results  of  this  analysis  are  shown  in  Figure  3.4.  Each  bar  repre¬ 
sents  the  total  number  of  priority  one  courses  funded  by  year  from  FY 
1998  through  FY  2005. 10  The  segments  of  each  bar  represent  the  status 
of  the  courses  as  of  the  end  of  the  first  quarter  of  FY  2008.  Conclusions 
from  this  analysis  are  discussed  below. 


10  With  perhaps  a  couple  of  exceptions,  all  courses  funded  in  FY  2006  and  after  were  still  in 
development  by  the  first  quarter  of  FY  2008. 
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Figure  3.4 

Number  of  Priority  One  DL  Courses  by  Graduate  Status  Category 
(FY 1998-2005) 
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Only  a  minority  of  priority  one  courses  funded  by  TADLP  ever 
produced  graduates.  While  an  average  of  about  28  courses  were  devel¬ 
oped  per  year,  only  an  average  of  7  of  these  have  had  graduates  (see 
the  bottom  segment  of  each  bar).  Most  of  the  remainder  of  the  courses 
were  completed  or  even  fielded,  but  no  graduates  were  recorded  in 
ATRRS.11 

In  addition  to  the  high  percentage  of  courses  that  were  apparently 
never  used,  several  efficiency  issues  also  arise  among  courses  that  had 
at  least  some  graduates. 

Many  successfully  developed  courses  did  not  last  in  the  field  past 
the  first  year.  From  FY  2004  through  FY  2006,  for  example,  up  to  half 
of  courses  with  graduates  or  requirements  “dropped  out”  one  year  later, 
that  is,  no  enrollments  or  requirements  for  that  course  appeared  the 


11  The  number  of  graduates  could  potentially  increase  for  the  years  that  still  have  courses 
outstanding,  especially  for  courses  funded  in  FY  2005.  Nineteen  fielded  courses  funded  in 
this  year  could  reasonably  still  show  graduates  before  becoming  obsolete,  and  another  17 
could  produce  graduates,  assuming  the  courses  are  completed.  However,  as  explained  below, 
experience  has  shown  that  courses  that  take  a  long  time  to  develop  also  become  obsolete 
quickly. 
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following  year.  FY  2007,  however,  showed  a  potential  improvement  in 
the  trend,  with  less  than  20  percent  of  courses  dropped  for  FY  2008. 

Fielded  DL  Courses  Have  a  Low  Number  of  Graduates 

DL  courses  tended  to  have  low  graduation  rates.  Residential  courses 
typically  have  completion  rates  over  90  percent,  (and  many  over  95 
percent).  While  not  strictly  comparable  due  to  differing  administrative 
processes  for  residential  courses,12  DL  modules  had  completion  rates 
in  FY  2006  of  50  to  67  percent,  depending  on  course  category.  MOS 
reclassification  and  BNCOC  had  completion  rates  at  the  low  end  of 
the  range — about  50  percent,  while  ANCOC  and  ASI/SQI  courses 
average  at  the  high  end.  CCC  courses  averaged  a  rate  of  about  60  per¬ 
cent.  While  somewhat  lower  graduation  rates  in  DL  courses  might  be 
accepted  as  an  inevitable  outcome  of  an  “any  where/any  time”  train¬ 
ing  strategy,  the  rates  shown  above  are  substantially  below  what  would 
likely  be  considered  an  acceptable  norm. 

The  average  number  of  graduates  from  DL  courses  was  relatively 
small;  in  fact,  most  courses  were  recorded  as  used  by  fewer  than  200 
graduates  in  FY  2006.  Average  course  graduation  among  active  DL 
courses  was  422.  Since  that  average  was  driven  by  a  few  courses  with 
graduates  in  the  thousands  (e.g.,  Battle  Staff  Course,  First  Sergeant 
Course,  and  BNCOC  Common  Core),  the  median  number  of  gradu¬ 
ates  was  smaller,  fewer  than  200  graduates  per  year.  Moreover,  30  per¬ 
cent  of  the  courses  produced  fewer  than  100  graduates  a  year.13 


12  Within  ATRRS,  enrollments  for  residential  courses  are  defined  as  those  students  who 
arrive  at  the  training  location  and  register.  Those  who  make  a  reservation  for  a  course  but 
do  not  show  up  are  measured  separately  as  “no  shows.”  For  DL  courses,  these  two  groups 
are  both  defined  as  enrollments.  Thus,  “graduation  rates”  for  DL  courses  have  a  somewhat 
different  meaning  than  they  do  for  residential  courses.  For  example,  if  a  student  has  a  health 
emergency  just  before  a  course  begins,  his  or  her  lack  of  participation  does  not  lower  the 
graduation  rate  in  residential  courses,  but  it  would  for  DL  courses. 

13  Higher  usage  rates  are  typical  for  DL  content  in  other  military  organizations.  For  exam¬ 
ple,  ATRRS  shows  that  the  Army  Materiel  Command  (AMC)  had  average  graduations  of 
750  per  course  hour,  Army  DL  courses  produced  outside  of  TRADOC  had  an  average  of 
2,750  graduations  per  course  hour,  and  DL  courses  within  the  Defense  Acquisition  Univer¬ 
sity  (DAU)  had  3,200  per  course  hour.  Note  that  these  results  are  measured  on  a  “per  course 
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Other  data  suggest  that  the  small  number  of  graduates  can  be 
traced  in  part  to  the  choice  of  courses  to  convert  to  DL.  An  efficient 
choice  of  content  would,  on  average,  imply  that  courses  with  higher 
student  load  would  be  chosen  for  conversion.  An  expected  higher  usage 
of  DL  content  would  allow  the  initial  cost  of  DL  development  to  be 
“amortized”  over  a  larger  number  of  trainees,  thus  yielding  a  larger 
ROI.  However,  data  show  that  for  four  of  the  five  course  categories 
considered,  student  load  for  DL  courses  was  no  greater  than  the  stu¬ 
dent  load  for  residential  courses.  The  only  exception  was  the  ASI/SQI 
category  of  courses,  where  DL  courses  had  an  average  of  nearly  1,200 
graduates  per  course  hour  in  FY  2006,  nearly  six  times  higher  than 
that  of  courses  in  that  category  not  converted  to  DL. 


Quality  of  Courseware  Output 

Given  the  Army’s  expectation  of  a  large  role  for  DL  in  training,  an 
important  component  of  TADLP’s  overall  performance  is  the  quality 
of  its  IMI  courses.  An  assessment  of  IMI  quality  is  necessary  not  only 
to  understand  and  create  a  baseline  assessment  of  what  the  Army  has 
received  for  its  investments  in  courseware  to  date,  but  also  to  make  the 
case  for  and  manage  budgets  to  make  needed  quality  improvements, 
and  to  identify  and  implement  needed  improvements  to  processes  that 
affect  quality. 

Approaches  to  Evaluation  of  Courseware  Quality 

The  most  popular  approach  to  training  evaluation  was  proposed  by 
Kirkpatrick  (1959,  1994),  who  identified  four  levels  of  training  out¬ 
comes  that  might  be  assessed  through  an  evaluation: 

•  Learner  reactions  typically  are  assessed  through  postcourse  sur¬ 
veys  of  student  satisfaction.  The  satisfaction  of  learners  is  the 
most  common  measure  of  reactions  (as  well  as  the  most  common 


hour”  rather  than  “per  course”  basis  in  order  to  account  for  the  variation  in  course  length 
across  categories. 
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method  of  evaluating  training).  Surveys  can  also  be  conducted  of 
other  groups,  such  as  instructors  or  other  school  personnel. 

•  Learning  refers  to  student  performance  in  the  training  program, 
which  can  be  assessed  by  measures  such  as  knowledge  tests  and 
skills  tests,  or  other  measures  such  as  peer  ratings  (Goldstein, 
1991).  Later  assessments  of  learning  (some  time  after  the  comple¬ 
tion  of  training)  can  be  used  to  assess  knowledge  retention. 

•  Behavior  reflects  whether  students  apply  what  they  learned  in 
training  on  the  job,  i.e.,  through  job  performance,  or  other  out¬ 
comes  such  as  rates  of  promotion  in  an  organization  (Goldstein, 
1991).  Behavior  is  also  referred  to  as  transfer  of  training.  Ideally, 
job  performance  is  measured  using  objective  criteria  (e.g.,  produc¬ 
tion  quality  or  time  to  complete  tasks).  Frequently,  however,  job 
performance  is  assessed  using  subjective  supervisory  performance 
ratings. 

•  Results  provide  information  about  the  effect  of  a  training  course 
or  program  on  organizational  outcomes.  For  example,  unit  readi¬ 
ness  might  be  used  as  a  measure  of  results.  However,  it  can  be 
difficult  to  identify  concrete  measures  of  organizational  perfor¬ 
mance  and  link  them  with  training. 

The  tests  associated  with  training  courses  might  also  be  evaluated. 
Several  methods  of  evaluating  training  effectiveness  depend  on  having 
good  tests.  There  are  both  qualitative  (content  validation)  and  quan¬ 
titative  or  statistical  (e.g.,  using  classical  test  theory  or  item  response 
theory)  methods  for  evaluating  tests. 

Another  approach  to  evaluating  training  quality,  and  the  focus  of 
our  companion  report  (Straus  et  ah,  2009),  is  to  assess  the  quality  of 
training  materials  using  external  evaluators.  In  a  courseware  content 
analysis,  evaluators  can  review  a  variety  of  characteristics  of  courses, 
such  as  the  comprehensiveness  of  the  course  content,  accuracy  of  infor¬ 
mation,  value  of  practical  exercises,  ease  of  use,  and  so  forth.  These 
criteria  are  especially  important  in  evaluating  DL,  as  less  is  known 
about  how  to  deliver  technology-mediated  instruction  compared  to 
traditional  in-person  training. 
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There  Is  No  Effective  Program-Wide  Evaluation  of  DL  Courseware 
Quality 

As  part  of  our  investigation  of  the  quality  of  DL  courseware,  we 
reviewed  existing  Army  regulations  (and  other  documentation)  regard¬ 
ing  evaluation,  and  we  interviewed  training  development  staff  from 
TRADOC  and  the  proponent  schools  with  regard  to  different  types 
of  training  evaluations  as  described  above.  We  also  conducted  a  pilot 
study  of  our  own,  a  content  analysis  of  IMI  courseware  available  to 
Army  learners  in  the  FY  2006-2007  time  frame.14  Our  conclusions 
follow. 

We  found  no  effective  evaluations  at  the  TADLP  level  (see  Table 
3.1).  AUTOGEN  is  the  only  common  system  for  course  quality  evalu¬ 
ation  across  TRADOC  schools.  AUTOGEN  is  a  program  for  design¬ 
ing  surveys  of  course  effectiveness  and  administering  them  to  course 
graduates  and  graduates’  supervisors.  Results  are  not  synthesized  across 
schools,  however,  and  for  reasons  described  in  detail  in  Chapter  Seven, 
AUTOGEN  does  not  provide  effective  assessments  of  DL  course  qual¬ 
ity.  The  responsibility  for  the  evaluation  of  DL  quality  is  decentral¬ 
ized  to  each  school  but  not  systematically  applied  across  all  schools. 
Moreover,  responses  to  interview  questions  also  showed  that,  in  gen¬ 
eral,  evaluations  of  DL  are  not  comprehensive  or  systematic,  in  terms 
of  either  the  range  of  training  evaluation  measures  collected  or  the 
standardization  of  measures  and  synthesis  of  results  across  courses  or 
schools  (see  Table  3.1). 

RAND  Arroyo  Center  Evaluation  of  Courseware  Quality 

To  shed  more  light  on  the  quality  of  TADLP  courseware,  we  con¬ 
ducted  a  content  evaluation  of  a  sample  of  lessons  from  ten  IMI  courses 
fielded  under  TADLP  between  2005  and  2007.  We  selected  a  sample 
of  courses,  stratified  by  proponent  school  and  course  level.  We  had 
online  access  to  two  of  the  courses,  and  the  remaining  eight  courses 
were  provided  on  CD.  Three  evaluators,  or  “coders,”  went  through 
the  courseware  in  much  the  same  way  a  student  would.  Typically,  we 


14  The  method  and  results  of  this  study  are  presented  in  detail  in  a  companion  document  to 
this  report  (see  Straus  et  ah,  2009). 
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Table  3.1 

Army  Evaluation  Efforts 


Type  of  Evaluation 

Does  Army  Conduct? 

Training  content 

Yes,  but  not  systematic 

Learner  reactions 

Some,  but  not  systematic 

Learning:  Pre/post  comparisons 

Yes,  but  not  systematic 

Learning:  Knowledge  retention 

No  known  efforts 

Behavior:  Predictive  validity 

No  DL-specific  efforts 

Results:  Unit  performance 

No  known  efforts 

Test  evaluation 

Yes,  but  not  systematic 

Indirect  metrics  (see  note) 

Yes,  but  limited 

NOTE:  The  term  "indirect  metrics"  refers  to  measures  of  effectiveness 
(e.g.,  DL  usage  or  cycle  time)  that  might  be  indirectly  related  to 
quality.  For  example,  if  DL  usage  is  low,  it  may  be  because  of  poor 
technical  or  content  quality  of  courseware. 


assessed  lessons  from  the  first  two  to  three  modules  of  each  course, 
where  we  define  a  module  as  a  general  topic  area  in  a  course  compris¬ 
ing  one  or  more  lessons.  (Note,  however,  that  terms  such  as  “module,” 
“lesson,”  and  “topic”  are  used  differently  by  the  proponent  schools.) 
In  some  cases,  we  selected  later  modules  in  the  courses  to  ensure  that 
we  evaluated  varied  content,  including  instruction  of  concepts,  pro¬ 
cesses,  and  procedures,  if  available.  In  total,  we  coded  79  lessons  that 
comprised  about  40  percent  of  the  lessons  per  course,  on  average,  or 
approximately  190  hours  of  instruction. 

We  did  not  examine  the  accuracy  or  comprehensiveness  of  the 
content,  but  rather  focused  on  the  instructional  design  and  technical 
aspects  of  the  courseware.  Criteria  for  the  evaluation  were  compiled 
and  adapted  from  key  sources  in  the  training  community,  including 
the  American  Society  of  Training  Development  (ASTD  Certification 
Institute,  2001-2003),  the  Center  for  Cognitive  Technology  at  the 
University  of  Southern  California,  and  TRADOC.  In  total,  we  rated 
courses  based  on  50  criteria  that  reflect  technical,  production-quality, 
and  pedagogical  features  of  courses.  Numerical  ratings  of  these  criteria 
for  the  courses  we  evaluated  can  be  found  in  Straus  et  al.  (2009).  We 
summarize  our  major  findings  here. 
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Although  based  on  a  small  sample,  content  analysis  points  to  some 
significant  issues  that  may  exist  with  regard  to  DL  courseware  quality. 
In  general,  although  a  number  of  courses  had  strengths  with  regard  to 
technical  features  and  production  quality,  the  assessment  found  serious 
shortcomings  in  pedagogy.  In  addition,  we  found  that  the  majority  of 
lessons  (76  percent)  were  classified  as  Level  2  IMI  or  less,  even  though 
the  contracts  in  use  at  the  time  called  for  Level  3  IMI. 

Technical  Features.  The  strongest  features  of  DL  courseware 
were  the  technical  characteristics  of  the  courses,  such  as  the  ease  of 
navigating  through  the  courseware.  We  identified  only  two  techni¬ 
cal  features  needing  substantial  improvement.  One  was  the  ability 
to  launch  courseware  without  technical  assistance.  Coders  required 
assistance  from  RAND  technical  support  to  launch  most  of  the 
courses  provided  on  CD.  If  Army  learners  experience  similar  prob¬ 
lems  without  access  to  technical  support,  it  is  reasonable  to  expect 
that  many  of  them  will  give  up,  which  would  be  reflected  in  low  com¬ 
pletion  rates.  The  second  technical  feature  needing  improvement  con¬ 
cerned  the  usability  of  supplementary  instructional  resources  (i.e.,  ref¬ 
erence  materials).  Although  most  courses  had  a  menu  through  which 
the  learner  could  easily  access  Field  Manuals  (FMs),  the  concepts  on 
any  particular  page  of  instruction  were  not  linked  directly  to  the  FM. 
Therefore,  using  these  resources  requires  substantial  effort  on  the  part 
of  the  learners. 

Production  Quality.  Production  quality  features  of  DL  course¬ 
ware,  i.e.,  the  ways  in  which  the  material  is  presented,  were  strong  in 
many  but  not  all  areas.  Narration  was  easy  to  understand,  courses  had 
minimal  irrelevant  content,  and  graphics  and  text  were  typically  leg¬ 
ible.  Most  courses  also  included  examples  of  effective  use  of  animation 
to  demonstrate  processes  and  procedures.  However,  some  aspects  of  the 
audiovisual  features  were  found  to  need  improvement.  Courses  were 
rated  as  only  “moderately  effective”  in  the  use  of  animation/video  to 
demonstrate  processes  and  procedures.  Narration,  while  easy  to  under¬ 
stand,  was  often  very  slow,  which  inhibited  the  pace  of  learning.  Sig¬ 
nificant  improvement  is  also  needed  in  using  media  to  engage  learners 
and  in  eliminating  sensory  conflicts. 
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Pedagogical  Features.  The  pedagogical  aspects  of  the  courseware 
were  the  most  importanr  criteria  in  our  quality  assessment.  Pedagogical 
features  refer  to  the  quality  of  the  instructional  content  and  processes. 
Pedagogical  strengths  of  the  courses  include  clear  descriptions  of  lesson 
objectives,  an  appropriate  order  of  lessons,  clear  and  comprehensive 
definitions  of  concepts,  and  opportunities  for  learners  to  correct  their 
strategies  in  checks  on  learning  and  practical  exercises. 

Pedagogy  was  the  area  most  in  need  of  improvement.  A  perva¬ 
sive  problem  in  many  of  the  courses  was  a  lack  of  examples  from  the 
job  or  mission  environments;  this  occurred  in  the  instruction  of  both 
concepts  and  prsocedures.  Courses  also  need  to  do  a  better  job  of  dem¬ 
onstrating  procedures  and  providing  explanations  of  why  procedures 
work  the  way  they  do,  so  that  learners  can  better  understand  the  con¬ 
cepts  and  skills  taught  and  can  thus  be  prepared  to  adapt  their  behavior 
in  nonstandard  situations.  Finally,  in  most  of  the  courses  we  evaluated, 
practical  exercises  did  not  provide  sufficient  opportunity  for  learners 
to  integrate  concepts  and  to  practice  procedures;  there  were  typically 
too  few  opportunities,  and  the  exercises  did  not  progress  from  simple 
to  complex  problems,  nor  did  they  include  both  part-task  and  whole- 
task  practice.  In  short,  the  courseware  was  deficient  with  respect  to  two 
critical  aspects  of  training:  effective  explanations  of  procedures  and 
opportunities  for  practice. 

While  recent  or  planned  changes  in  TADLP  processes  (discussed 
later)  may  have  increased  the  quality  of  Army  IMI  for  courses  under 
development,  most  of  these  products  were  not  available  to  learners  in 
the  FY  2006  time  frame. 


Cycle  Time  for  Courseware  Production 

We  now  turn  to  the  results  of  our  analysis  of  the  DL  production  pro¬ 
cesses.  In  particular,  we  examined  the  amount  of  time  required  to  pro¬ 
duce  a  DL  course  in  the  Army  and  in  the  commercial  sector.  Cycle 
time  directly  affects  the  efficiency  of  the  production  process  and  the 
responsiveness  of  DL  to  changing  Army  training  needs.  Moreover, 
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cycle  time  indirectly  affects  courseware  quality  and  value  because  the 
currency  of  the  course  decreases  over  time. 

As  discussed  in  Chapter  Two,  we  define  a  complete  cycle  in  the 
courseware  development  process  as  the  period  between  the  identi¬ 
fication  of  the  DL  requirement  and  the  time  students  begin  to  use 
the  product.  To  get  a  better  perspective  on  the  cycle  time  of  TADLP 
courseware,  we  compared  the  end-to-end  time  for  development  in  the 
Army  to  the  private  sector.  To  understand  the  results  of  this  compari¬ 
son  more  clearly,  we  broke  the  cycle  time  of  the  commercial  process  into 
the  same  two  major  parts  as  we  divided  the  Army  process  described  in 
Chapter  Two:  the  time  from  the  identification  of  the  need  to  the  sign¬ 
ing  of  the  contract,  and  the  time  from  contract  signing  to  the  first  use 
of  the  training  content. 

Commercial  courses  are  often  shorter  than  the  multi-week  courses 
designed  by  the  Army  (multi-hour  as  opposed  to  multi-week).  Thus,  in 
our  initial  effort,  we  inquired  about  the  time  it  would  take  for  a  typical 
10-hour  commercially  developed  effort,  and  for  a  50-hour  effort  within 
TADLP.  Then,  to  make  the  two  periods  more  comparable,  we  sepa¬ 
rately  estimated  the  time  needed  for  a  commercial  company  to  expand 
its  effort  to  a  full  50  hours.  This  estimate  was  less  reliable  than  the  first, 
as  the  longer  course  is  atypical  for  commercial  projects  but  was  judged 
sufficient  for  broad  comparisons.15 

Figure  3.5  shows  the  results  of  our  analysis,  and  leads  to  the  fol¬ 
lowing  conclusions. 

The  Army  takes  significantly  longer  to  develop  IMI  than  do  com¬ 
mercial  firms.  Our  analysis  found  that  to  develop  50  hours  of  Level  2 
and  Level  3  IMI,  the  Army  took  over  three  and  a  half  years,  nearly  four 
times  as  long  as  commercial  firms,  which  took  less  than  a  year  (Figure 

3.5). 

Cycle  times  in  the  commercial  sector  were  solicited  during  inter¬ 
views  with  several  managers  of  commercial  training  development  orga¬ 
nizations.  They  indicated  it  would  take  less  than  two  months  to  secure 


15  Commercial  firms  stated  that  expanding  their  effort  from  10  to  50  hours  would  only  add 
a  few  months  to  the  development  time,  due  to  scale  economies  and  their  ability  to  quickly 
expand  (or  contract)  their  effort  to  fit  customer  need. 
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Figure  3.5 

Months  Needed  to  Develop  Level  2  and  Level  3  IMI 
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a  contract,  and  about  9  months  for  development  (Figure  3.5).  Data 
were  collected  in  two  stages.  First,  we  asked  firms  to  estimate  the  time 
it  would  take  to  produce  10  hours  of  IMI  (a  more  typical  length  for 
the  commercial  sector).  In  the  second  stage,  we  asked  those  with  some 
experience  of  large  contracts  to  estimate  how  much  more  time  it  would 
take  to  develop  40  more  hours  after  the  first  10.  All  interviewees  indi¬ 
cated  they  would  hire  extra  personnel  for  a  surge  effort  to  complete  the 
extra  hours. 

Cycle  times  in  the  Army  were  estimated  for  FY  2006  from  actual 
experience,  partially  calculated  via  ATSC  records,  and  partially  esti¬ 
mated  through  interviews  of  school  personnel.  The  analysis  concluded 
it  would  take  about  18  months  to  secure  a  contract,  and  about  another 
26  months  for  soldiers  to  begin  taking  the  course  (Figure  3.5). 

The  estimate  of  Army  IMI  development  times  in  Figure  3.5 
includes: 
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•  An  assumed  average  of  6  months  from  first  identification  of 
need  for  a  training  IMI  product  to  submission  of  a  “nomination” 
request  to  the  TRADOC  DL  courseware  process.16 

•  5.5  months  from  official  nomination  by  the  end  of  the  fiscal  year 
to  approval  of  the  course  in  mid-March.17 

•  6.5  months  from  the  decision  to  fund  the  course  to  signing  a 
contract  by  the  end  of  the  next  fiscal  year.18  The  first  three  steps 
together  constituted  the  18 -month  “need  to  contract”  time  shown 
in  Figure  3.5. 

•  Actual  average  time  from  contract  start  at  the  beginning  of  a  fiscal 
year  (i.e.,  in  October)  to  completion  of  the  courseware  (including 
all  testing),  based  on  FY  2005  and  FY  2006  courseware  comple¬ 
tion  data  was  about  24  months.19  This  number  comes  from  the 
actual  average  time  for  FY  2005  courseware  that  is  complete.20 

•  An  estimate  of  two  months  from  final  delivery  of  the  course  to 
use  by  soldiers  (included  as  part  of  the  solid  gray  bar).  The  last  two 
steps  together  constituted  the  26-month  “contract  to  use”  time 
shown  in  Figure  3.5. 

The  biggest  difference  between  the  Army  and  commercial  firms 
was  in  the  time  segment  from  “need  to  contract,”  which  took  more 


16  The  assumption  is  that  the  need  on  the  part  of  the  proponent  school  could  be  identi¬ 
fied  either  just  before  the  deadline  for  submission,  or  up  to  12  months  before  the  deadline; 
averaging  the  two  numbers  yields  the  6-month  estimate.  Note  that  this  calculation  does  not 
account  for  Army  policy  regarding  identifying  the  need  for  DL,  which  is  not  uniformly  fol¬ 
lowed.  TRADOC  Regulation  350-70,  Chapter  II-8,  calls  for  identification  of  need,  and  the 
filing  of  Course  Administrative  Data,  three  years  before  the  funding  of  the  course. 

17  The  deadline  for  submissions  is  the  end  of  September  each  year,  corresponding  to  the  end 
of  the  fiscal  year.  The  annual  meeting  for  selection  of  DL  courseware  for  development  is  in 
mid-March  of  the  following  year,  or  5.5  months  later. 

18  The  funding  of  the  DL  courseware  development  has  typically  taken  place  in  September  of 
the  same  fiscal  year,  so  from  mid-March  to  the  end  of  September  is  6.5  months. 

19  Even  though  contracts  are  written  as  one-year  agreements,  many  courses  receive  “no  cost” 
extensions  that  lengthen  the  contract  period.  In  addition,  the  time  needed  for  courseware 
testing  at  ATSC  comes  after  the  contracting  period. 

20  Acceptance  of  ATSC  projections  of  further  reductions  in  cycle  time  reduced  this  number 
by  6  months. 
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than  10  times  as  long  in  the  Army  (1.75  months  versus  18  months). 
The  commercial  companies  are  not  often  constrained  by  annual  cycles, 
but  instead  create  new  content  when  it  is  needed.  They  are  also  adept 
at  setting  a  case-by-case  contracting  period  suited  to  customer  needs 
and  their  own  capabilities  and,  when  necessary,  contracting  for  short 
periods  of  time  to  produce  content  quickly. 

We  also  attempted  to  look  at  trends  in  cycle  time  to  estimate 
future  trends.  The  available  administrative  data  did  not  allow  us  to 
examine  the  full  cycle  time,  but  only  the  time  from  the  date  of  the  con¬ 
tract  to  the  actual  (or  estimated)  time  of  fielding,  which  is  about  two 
months  before  actual  use.  Trend  results  for  this  period  are  presented  in 
Figure  3.6,  which  tracks  actual  development  time  for  courses  within 
TADLP  funded  from  FY  1998  to  FY  2005,  and  estimates  that  develop¬ 
ment  time  for  courses  funded  in  FY  2006.  The  graph  shows  not  only 
the  average  times,  but  also  median  and  75th  percentile  times,  which 
provide  a  sense  of  the  variance  and  outliers  in  times.  We  draw  the  fol¬ 
lowing  conclusions: 

Long  cycle  times  from  contract  to  fielding  are  nothing  new  in 
the  Army;  in  fact,  times  were  longer  in  the  years  preceding  the  period 
evaluated.  Even  longer  cycle  times  than  shown  in  the  previous  chart 
were  common  nearer  the  beginning  of  TADLP;  in  particular,  between 
FY  1998  and  FY  2003,  the  average  yearly  development  time  ranged 
between  40  and  50  months. 

Development  times  (from  contract  to  fielding)  were  significantly 
reduced  in  the  latter  part  of  the  period.  Improvements  began  in  FY 
2004  and  continued  through  FY  2005,  where  the  average  number  of 
months  fell  to  below  30.  The  shortest  documented  time,  24  months, 
occurred  for  courses  funded  in  FY  2005;  this  was  the  number  used 
on  the  previous  chart,  where  the  Army  was  compared  to  commercial 
firms.  ATSC  projects  that  the  time  between  funding  and  fielding  for 
FY  2006  courses  will  approach  about  a  year  and  a  half,  but  those  times 
could  not  be  verified  before  completion  of  this  report. 


The  State  of  Army  DL  55 


Figure  3.6 

Army  DL  Cycle  Time  (FY  1998-2006) 


Time  between  year  of  funding  and  actual  or  projected  date  of  fielding  DL  products 


SOURCE:  ATSC. 

NOTE:  FY06  downward  trend  based  on  ATSC  projections. 
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Responsiveness  of  the  Process  for  Producing  Courseware 

We  now  discuss  the  responsiveness  of  courseware,  which  we  define  as 
the  ease  with  which  courseware  can  be  adapted  in  response  to  chang¬ 
ing  requirements.  To  examine  how  responsive,  or  adaptive,  TADLP 
processes  were  (by  the  end  of  FY  2006)  in  effecting  timely  change  to 
courseware  products,  we  focused  on  three  periods  through  the  lifecycle 
of  IMI  content:  the  period  from  requirement  identification  to  signing 
of  a  contract,  the  period  of  courseware  development,  and  the  post¬ 
production  period.  While  there  is  some  overlap  in  this  measure  with 
cycle  time  (discussed  above),  the  focus  here  will  be  on  how  quickly  the 
Army  can  turn  a  new  requirement  into  IMI  courseware  or  change  an 
existing  idea  without  sacrificing  training  quality  or  other  measures  of 
effectiveness. 

IMI  courseware  is  relatively  unresponsive  to  urgent  or  changing 
requirements  before,  during,  and  after  initial  development.  Starting 
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from  our  discussion  above  on  cycle  time,  when  the  idea  of  an  IMI  prod¬ 
uct  first  emerges,  the  TADLP  process  does  not  allow  for  a  rapid  path 
to  a  contract  to  produce  IMI-based  training.  In  the  commercial  sector, 
firms  can  go  from  “idea  to  contract”  within  weeks  or  a  few  months.  In 
the  Army,  the  average  time  has  been  over  a  year.  IMI  ideas  are  essen¬ 
tially  “batched”  in  the  Army,  going  into  a  pool  for  consideration  and 
potential  selection  at  a  prioritization  conference  that  convenes  once  a 
year.  Then  funding  is  typically  further  delayed  when  funds  are  not 
received  until  the  end  of  one  fiscal  year  or  the  beginning  of  the  next. 

While  changing  an  idea  before  a  contract  is  signed  is  possible,  it 
can  often  take  a  long  time  to  implement.  Funding  a  nominated  prod¬ 
uct  requires  complete  GFI/GFM;  these  typically  are  not  readily  avail¬ 
able  and  often  take  considerable  time  to  complete  initially  or  to  change 
midway  through  the  process  because  of  the  shortage  of  TRADOC 
training  development  resources.  GFI  and  GFM  can  include  an  exten¬ 
sive  list  of  documents,  equipment,  software,  facilities,  and  services. 
For  example,  they  can  include  the  program  of  instruction  (POI),  field 
manuals,  regulations,  lesson  plans,  course  management  plan,  approved 
critical  tasks  and  task  analysis  data,  graphic  training  aids,  maps,  art¬ 
work,  and  video  presentations. 

Once  IMI  content  begins  development  in  the  Army,  it  is  difficult 
to  change  the  requirement  without  significant  penalty.  In  the  commer¬ 
cial  sector,  agreements  for  IMI  content  can  and  are  routinely  changed 
as  development  proceeds.  In  contrast,  the  Army’s  product-focused, 
exclusively  fixed-price  contracting  vehicle  makes  it  almost  impossible 
to  make  formal  changes  to  the  contractor’s  initial  “statement  of  work.” 
Because  there  is  often  a  need  to  change  IMI  requirements  in  the  Army 
context,  parties  have  often  found  a  way  to  informally  alter  what  IMI 
content  is  produced,  but  the  change  typically  comes  at  the  sacrifice  of 
another  part  of  the  requirement  or  timely  completion  of  the  product. 

Once  a  course  is  fielded  to  the  Army,  new  development  to  make 
necessary  changes  in  content  has  typically  not  been  possible  for  a 
number  of  years,  even  for  development  efforts  that  would  require  only 
weeks  or  months.  Again,  no  such  constraints  exist  in  the  commer¬ 
cial  sector.  In  the  Army  context,  such  changes  have  been  treated  as  a 
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request  for  new  content,  involving  the  same  waiting  period  to  compete 
for  prioritization  and  funding. 


Chapter  Conclusions 

The  findings  presented  in  this  chapter  indicate  that,  after  nine  years 
of  implementation  and  a  growing  need  for  DL  across  the  Army,  the 
impact  of  TADLP  is  still  relatively  minor.  There  are  clear  highlights  in 
the  program,  such  as  high  use  of  some  courses  (especially  in  the  ASI/ 
SQI  category)  and  favorable  hourly  rates  for  courseware  production. 
However,  in  terms  of  our  primary  criterion — program  impact — the 
amount  of  DL  training  executed  is  a  small  proportion  of  all  training, 
especially  for  the  AC,  and  the  content  of  the  training  is  relatively  basic. 

Evaluated  using  our  measures  of  quality,  efficiency,  cycle  time, 
and  responsiveness,  TADLP’s  relatively  small  courseware  program 
does  not  fare  well.  The  typically  short  lifespan  of  courseware  once  it 
becomes  active  and  low  usage  rates  lead  to  a  small  return  on  investment 
on  the  funds  expended,  despite  the  relatively  low  hourly  costs  the  Army 
has  been  able  to  negotiate  with  contractors.  Further,  although  we  were 
not  able  to  examine  all  aspects  of  courseware  quality,  our  sample  evalu¬ 
ation  suggests  areas  of  deficiency,  especially  in  the  pedagogy  employed. 
Finally,  long  cycle  times  for  production,  as  well  as  the  difficulties  of  get¬ 
ting  new  ideas  into  production,  making  changes  once  production  has 
commenced,  and  making  revisions  to  existing  courses,  all  add  up  to  a 
product  and  process  that  have  an  insufficient  capability  for  adapting  to 
and  supporting  the  Army’s  rapidly  increasing  and  changing  needs  for 
course  content. 


CHAPTER  FOUR 


Improvements  to  the  IMI  Program 


The  previous  chapter  identified  a  number  of  areas  in  need  of  improve¬ 
ment  in  TADLP’s  courseware  program  for  IMI.  In  this  chapter  we 
consider  some  of  the  underlying  issues  that  led  to  the  outcomes 
described  in  Chapter  Three.  We  also  present  a  number  of  potential 
near-term  improvements  to  the  IMI  program  that  can  positively  influ¬ 
ence  TADLP  outcomes  with  regard  to  the  measures  of  effectiveness 
presented  in  Chapter  Three. 

Five  improvement  initiatives  are  proposed  in  this  chapter  that 
focus  on  the  Army’s  approach  to  IMI  in  the  FY  2007-2008  time 
frame.  Broader,  more  strategic  initiatives  for  TADLP  are  considered  in 
subsequent  chapters.  These  five  initiatives  cover  the  span  of  courseware 
development,  from  design  through  delivery: 

•  Design  of  program: 

-  Add  flexibility  to  the  courseware  acquisition  strategy  (e.g.,  by 
decentralizing  contract  management  and  administration  in 
appropriate  ways)  to  achieve  a  range  of  overall  improvements. 

-  Allocate  sufficient  resources  per  training  module  for  stand¬ 
alone  IMI  (even  if,  in  the  short  term,  this  means  funding  fewer 
courses)  to  promote  efficiency,  shorten  cycle  time,  and  improve 
quality  and  responsiveness. 

•  Before  and  during  development: 

-  Undertake  systematic  process  improvements  to  reduce  cycle 
time. 
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-  Increase  local  participation  in  IMI  production  and  contract 
administration  to  reduce  cycle  time  and  increase  efficiency, 
quality,  and  responsiveness. 

•  After  development  and  during  delivery: 

-  Institute  a  program-level  IMI  evaluation  component  to  improve 
quality  and  other  measures  of  effectiveness. 

For  each  area  of  improvement,  we  first  highlight  what  we  learned  about 
the  problem,  and  then  make  recommendations  for  improvement.  It 
should  be  noted  that  these  recommendations  are  intended  to  cut  across 
the  issues  identified  in  Chapter  Three,  not  to  track  them  one-to-one. 
The  likely  effects  of  each  major  recommendation  on  the  measures  of 
effectiveness  presented  in  Chapter  Three  are  summarized  in  Table  4.8 
at  the  end  of  this  chapter. 


Add  Flexibility  to  the  Courseware  Acquisition  Strategy 

One  of  the  findings  discussed  in  Chapter  Three  concerned  the  general 
unresponsiveness  of  TADLP  courseware  to  changing  or  newly  urgent 
requirements.  This  problem  is  particularly  apparent  in  the  acquisition 
process  used  for  IMI  courseware,  which  is  the  first  area  of  improve¬ 
ment  discussed  in  this  chapter. 

We  recommend  a  number  of  near-term  and  longer-term  changes 
to  the  Army’s  IMI  acquisition  strategy  to  increase  TADLP  responsive¬ 
ness.1  Further,  we  argue  that  these  changes  would  also  shorten  cycle 
times  and  improve  efficiency  and  quality.  To  provide  context  for  under¬ 
standing  the  recommendations,  we  first  briefly  discuss  the  Army’s 
approach  to  acquisition  strategy  for  IMI. 

Issues  Regarding  the  Army's  Acquisition  Strategy  for  IMI 

The  Army's  acquisition  strategy  is  composed  of  three  elements. 

The  choices  the  Army  has  made  regarding  these  key  elements  of  its 


1  For  a  more  detailed  explanation,  see  a  separate  unpublished  draft  by  Coombs  and  Shanley 
(2008). 
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acquisition  strategy  for  IMI  have  contributed  to  a  lack  of  flexibility 
and  overall  effectiveness  in  TADLP  implementation.  These  elements 
involve  the  Army’s  characterization  of  the  IMI  output  (courseware), 
its  approach  to  purchasing  courseware  content,  and  the  design  of  the 
contract  vehicle  used  to  acquire  courseware.  While  geared  toward  low- 
cost  purchases,  the  Army  acquisition  strategy  is  not  flexible  enough  to 
address  many  of  its  IMI  requirements  as  well  as  a  number  of  its  key 
constraints. 

First,  the  Army  designates  IMI  courseware  as  a  product,  a  charac¬ 
terization  more  appropriate  for  items  that  are  not  expected  to  change. 
In  acquisitions,  the  output  of  the  acquisition  process  must  be  catego¬ 
rized  as  a  product,  service,  or  system.  A  product  characterization  is  typi¬ 
cally  given  when  the  output  of  the  process  is  tangible  and  not  expected 
to  change  during  the  procurement  period.  In  contrast,  a  service  char¬ 
acterization  is  more  appropriate  when  tasks  are  not  as  well  defined, 
and  a  system  characterization  is  appropriate  when  the  final  product 
is  expected  to  be  greater  than  the  sum  of  its  parts,  i.e.,  the  capabil¬ 
ity  will  be  derived  from  a  possibly  complex  combination  of  products 
and  services  that  are  potentially  subject  to  significant  change  over  the 
development  period.  Although  IMI  courseware  typically  needs  ongo¬ 
ing  updates  and  changes  in  response  to  changing  requirements,  the 
Army  characterizes  the  courseware  as  a  product,  so  from  the  start,  the 
process  is  designed  not  to  incorporate  revisions  either  during  or  after 
development. 

Second,  the  Army  uses  a  “single-step”  acquisition  approach.  The 
inflexibility  of  the  Army’s  acquisition  process  for  IMI  is  also  due  to  the 
single-step  approach  used.  Such  an  approach  focuses  on  making  a  pur¬ 
chase  quickly  and  for  a  fair  price,  but  it  does  not  include  any  built-in 
mechanisms  to  deal  with  changing  requirements. 

An  alternative  to  the  single-step  approach  is  the  evolutionary 
approach ,  which  takes  place  over  time.  One  example  of  an  evolution¬ 
ary  approach  is  an  incremental  development  approach ,  which  is  typically 
used  when  the  requirement  itself  is  relatively  well  known  at  the  outset, 
but  the  associated  information,  resources,  and  available  technology  rel¬ 
evant  to  that  requirement  are  expected  to  change  over  the  product  life- 
cycle.  Another  example  is  a  spiral  development  approach,  which  is  used 


62  Making  Improvements  to  The  Army  Distributed  Learning  Program 


when  even  the  final  requirements  are  not  well  known  at  the  time  of 
program  initiation. 

Finally,  the  Army  uses  a  firm  fixed-price  contract  type  for  IMI, 
which  assumes  that  all  requirements  are  clearly  known  in  advance.  The 
firm  fixed-price  contract  is  used  in  the  Army’s  Distributed  Learning 
Education  and  Training  Products  (DLETP)  contract  suite,  a  group  of 
six  “multiple  award,  delivery  order  contracts”  (MADOCs).  A  fixed- 
price  contract  provides  for  firm  pricing  and  a  well-defined  compliance 
arrangement;  typically,  the  contractor  agrees  to  provide  the  output 
regardless  of  difficulties  that  might  arise  during  development.  Fixed- 
price  contracts,  as  the  name  suggests,  are  designed  to  resist  changes  in 
price. 

The  Army  has  several  other  options  for  contracting.  For  example, 
cost  contracts  provide  for  reimbursement  of  a  contractor’s  reasonable 
costs,  whether  or  not  the  expenditures  result  in  an  output  that  fulfills 
the  requirement.  Cost  contracts  are  typically  used  when  the  customer 
is  unclear  about  the  details  of  the  requirement,  and  is  willing  to  pay 
the  contractor  to  collaborate  and  innovate  in  order  to  get  to  the  specific 
solution.  Other  types  of  contracts  include  time  and  materials,  require¬ 
ments,  fixed-price  multiple  award  with  time  and  materials  line  item, 
and  fixed-price  with  prospective  price  redetermination. 

The  acquisition  framework  is  not  flexible  enough,  leading  to 
adverse  impacts  on  multiple  measures  of  effectiveness  for  TADLP. 
The  Army’s  acquisition  framework  is  problematic  for  several  reasons. 

First,  the  presumed  level  of  upfront  specificity  built  into  the  acqui¬ 
sition  strategy  often  cannot  be  achieved  in  practice,  resulting  in  adverse 
effects  on  cycle  time  and  courseware  quality.  The  reason  upfront  speci¬ 
ficity  cannot  be  achieved  is  that  requirements  for  most  Army  course¬ 
ware  change  frequently  over  the  development  period,  after  the  delivery 
order  (DO)  is  signed.  Changes  derive  partly  from  changes  in  content 
and  the  design  of  the  overall  course  that  IMI  phases  support  and  partly 
from  changes  with  military  occupational  specialties — realities  that  are 
becoming  more  common  over  time.  But  even  when  content  areas  are 
expected  to  be  relatively  stable  (the  Army  tries  to  find  such  cases  when 
it  nominates  content  for  funding),  incompletely  developed  GFI/GFM 
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often  lead  to  ill-defined  requirements  and  the  need  for  substantial 
changes  during  development. 

While  the  effects  of  anticipated  change  can  sometimes  be  miti¬ 
gated  by  better  upfront  coordination,  the  acquisition  framework  effec¬ 
tively  bars  Army  schools  and  contractors  from  discussing  and  negotiat¬ 
ing  specifics  of  proposals  before  DOs  are  signed.  Under  its  acquisition 
strategy,  the  Army  competes  every  DO  among  all  its  prime  contractors. 
In  this  context,  discussions  with  contractors  prior  to  the  DO  would 
represent  an  administrative  burden  because,  to  maintain  fairness  of  the 
competition,  discussions  would  have  to  involve  all  six  contractors  in 
every  case.  Moreover,  the  time  between  solicitation  and  award  is  short 
(often  only  ten  days),  both  because  fixed-price  product  contracts  are 
supposed  to  be  well  defined  enough  not  to  need  further  negotiation, 
and  because  ATSC  typically  receives  its  money  for  courseware  develop¬ 
ment  with  little  time  remaining  in  the  fiscal  year  to  award  it. 

The  reality  of  change  combined  with  an  inflexible  framework 
leads  to  increases  in  cycle  time  and  decreases  in  quality.  Both  schools 
and  contractors  have  reported  substantial  obstacles  in  making  satis¬ 
factory  adjustments  to  IMI  requirements  once  a  DO  is  signed.  (See 
Appendix  B,  in  particular  the  section  discussing  school-contractor 
interaction.)  Moreover,  the  informal  methods  typically  used  to  make 
change  possible  in  practice  often  detract  from  the  effectiveness  of  the 
content  being  produced.  For  example,  contractors  and  schools  some¬ 
times  negotiate  informally  to  downgrade  capabilities  (e.g.,  lower  inter¬ 
activity  levels  or  less  content  coverage)  in  some  areas  in  order  to  accom¬ 
modate  new  requirements  in  other  areas.  Further,  in  lieu  of  changes 
that  would  require  an  increase  in  the  price  of  an  effort,  no-cost  exten¬ 
sions  are  often  granted,  which  adversely  impact  on  the  cycle  time  of 
IMI  production. 

Courseware  maintenance  and  revision  procedures  generate  addi¬ 
tional  cycle  time  challenges  that  affect  course  quality  and  program 
efficiency.  The  “single  step”  and  “product”  approaches  built  into  the 
Army  strategy  basically  assume  that  revisions  will  not  be  part  of  the 
requirement.  As  a  result,  maintenance  actions  on  existing  courses  in 
the  Army  case  are  typically  treated  as  if  they  were  new  acquisitions, 
requiring  a  new  nomination  and  selection  process  and  a  new  competi- 
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tion  for  the  new  work  among  all  the  contractors.  These  processes  take 
years  to  implement.  As  a  result,  schools  identified  “maintenance”  as  a 
major  obstacle  in  the  success  of  TADLP  (see  Appendix  B  for  specifics). 
In  practice,  schools  often  delay  revisions,  leading  to  outdated  course¬ 
ware,  or  remove  the  courseware,  resulting  in  less  DL  contribution  to 
training. 

Process  issues  surrounding  acquisition  also  adversely  affect 
quality,  cycle  time,  and  efficiency.  Two  process  issues  related  to  the 
acquisition  strategy  also  warrant  discussion.  First,  the  criteria  used  in 
the  evaluation  and  award  process  are  too  broad  for  selecting  the  best 
contractor  and  for  taking  into  account  individual  school  requirements. 
As  a  result,  DOs  nearly  always  go  to  the  lowest  bidder,  which  will  not 
always  be  the  most  cost-effective  choice  for  the  government.  It  is  likely 
that  a  sizable  number  of  problems  identified  by  schools  and  contractors 
in  their  interaction  (see  Appendix  B)  derive  from  this  tendency. 

Second,  the  TRADOC  practice  of  funding  IMI  at  the  end  of  the 
fiscal  year  adds  to  the  cycle  time  by  nearly  a  year.  The  funding  surges 
that  generally  occur  at  the  end  of  the  year  lead  to  simultaneous  compe¬ 
tition  of  dozens  of  DOs,  ultimately  degrading  the  quality  of  the  DOs 
that  govern  IMI  development.  For  example,  42  DOs  were  competed  in 
the  final  month  of  FY  2007,  which  could  have  led  to  as  many  as  252 
proposals  to  consider  if  each  prime  contractor  bid  on  every  DO.  This 
kind  of  surge  inevitably  places  pressure  on  the  schools,  contractors,  and 
government  staff,  leading  to  inevitable  errors  and  shortcuts  in  processes 
that  deserve  more  careful  preparation.  To  the  degree  that  the  resulting 
DOs  do  not  outline  the  customer’s  requirements,  the  responsiveness 
and  speed  of  delivery  will  be  affected. 

Recommendations  for  Improvement 

We  propose  a  both  a  near-term  and  a  longer-term  approach  to  address 
the  shortcomings  of  the  acquisition  strategy  for  IMI. 

Make  near-term  improvements  to  the  acquisition  approach.  In 
the  near  term,  we  recommend  making  improvements  that  better  suit 
the  Army’s  requirements  and  constraints.  Table  4.1  lists  the  issues  that 
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Table  4.1 

Potential  Near-Term  Changes  to  the  Army's  Acquisition  Strategy 


Issue 

Potential  Change 

Need  to  change  content  during 
development 

Add  service  tasks  to  contract 

Need  to  maintain  and  revise  content 
after  development 

Adopt  practice  of  "logical  follow-on" 
orders 

Choose  contractor  with  "best  value,"  not 
lowest  price 

Revise  evaluation  and  award  process 
Eliminate  discounted  bids 

can  be  addressed  in  the  near  term  (in  the  left  column)  and  describes  the 
potential  changes  to  be  made  (in  the  right  column).2 

To  address  the  need  for  more-responsive  changes  in  training  con¬ 
tent  during  the  development  period,  the  Army  could  add  service  tasks 
to  the  contract  via  a  “time  and  materials  Contract  Line  Item  Number 
(CLIN).”  The  CLIN,  which  was  not  legal  at  the  time  the  original 
DLETP  was  designed,  would  allow  changes  (e.g.,  modifications  to  the 
DO  or  an  upgrade  in  technology)  to  be  specified  and  implemented 
after  the  finalization  of  the  DO.3 

To  address  the  need  to  maintain  and  revise  content  after  devel¬ 
opment,  the  Army  could  liberalize  the  use  of  logical  follow-on  orders, 
including  orders  above  $100,000.  Use  of  these  orders  could  greatly 
reduce  the  time  it  takes  to  implement  a  maintenance  DO,  and  it  would 
foster  greater  efficiency  and  quality  as  well.  Further,  this  practice  would 
allow  more  DL  courseware  to  stay  active,  increasing  the  impact  of  DL 
on  training  and  thus,  ultimately,  ROI. 

To  improve  school-contractor  relations  and  further  foster  higher- 
quality  output,  the  Army  could  initiate  a  revision  of  the  contract  eval¬ 
uation  and  award  process  to  facilitate  the  selection  of  the  contractor 
that  represents  the  “best  value”  for  the  government.  Under  the  new 


2  For  expanded  explanations  and  additional  suggestions  on  near-term  changes,  see  the  sep¬ 
arate  unpublished  draft  by  Coombs  and  Shanley  (2008). 

3  The  Army  has  recently  improved  its  contracting  process  by  allowing  (via  what  is  called  an 
“enhancements  CLIN”)  nonstandard  tasks  to  be  added  to  the  delivery  order.  However,  this 
practice  is  less  flexible  than  what  is  proposed  here  in  that  it  requires  that  the  enhancement  be 
fully  defined  prior  to  the  signing  of  the  DO.  No  services  after  signing  are  allowed. 
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process,  greater  weight  would  be  given  to  factors  relating  to  superior 
performance,  such  as  the  contractor’s  expertise  in  a  school’s  occupa¬ 
tional  area,  or  the  contractor’s  past  performance  with  regard  to  quality, 
timeliness,  and  cost  control.  To  implement  the  change,  input  would  be 
required  from  ATSC  and  TRADOC  staff,  TRADOC’s  contracting 
officer  for  the  current  contract,  and  the  proponent  schools. 

Develop  a  new  acquisition  strategy  for  IMI  that  better  suits  the 
Army's  needs  for  longer-term  improvement.  While  modifying  the 
acquisition  strategy  can  increase  efficiency  and  effectiveness  in  the  near 
term,  additional  benefits  are  possible  if  the  Army  begins  to  develop  a 
new  long-term  acquisition  strategy  now,  before  the  five-year  DLETP 
contract  expires.  The  new  strategy  would  be  designed  to  better  accom¬ 
modate  the  need  for  flexibility  to  meet  Army  needs.  The  new  strategy 
would  have  three  main  components:  requirements  contracts,  selective 
use  of  system  output  designation,  and  selective  use  of  an  incremental 
acquisition  approach. 

Adopt  requirements  contracts.  We  recommend  that  the  Army 
adopt  a  strategy  centered  around  requirements  contracts ,  which  would 
“prequalify”  a  set  of  prime  contractors  for  DL  development  to  work 
within  separate  well-defined  areas.  Under  such  an  arrangement,  several 
contractors  would  each  earn  a  separate  requirements  contract  for  any 
future  requirements  that  fall  within  each  unique  scope  of  work  over 
specified  periods  (usually  five  years).  This  would  eliminate  the  need 
to  compete  each  DO  separately.  For  example,  one  contractor  might 
develop  all  medical  DL  courseware.  Courseware  requirements  could 
be  grouped  by  proponent  or  geographical  location.  The  new  contract 
would  be  structured  to  provide  flexibility  depending  on  the  nature  of 
the  requirement,  and  it  could  have  elements  that  provide  incentives  for 
efficiency  and  quality.  A  fixed-price  CLIN  could  be  incorporated  to 
fund  a  front-end  analysis,  which  would  allow  both  the  Army  and  con¬ 
tractors  to  develop  a  better  understanding  of  the  requirement  before 
committing  sizable  resources  on  an  unclear  effort.  Performance  incen¬ 
tives,  which  schools  noted  were  often  lacking  for  IMI  contracts  (see 
Appendix  B),  should  also  be  built  into  the  contract,  including  those 
that  could  extend  the  contract  term. 
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Make  selective  use  of  systems  output  designation.  The  new 

strategy  would  also  selectively  use  a  “systems  output”  philosophy 
during  acquisition  planning.  Parallel  to  commercial  software  procure¬ 
ments,  this  would  mean  that  the  entire  lifecycle  of  the  course  is  consid¬ 
ered  during  the  acquisition  process,  and  that  total  ownership  costs — 
not  just  development  costs — are  taken  into  account.  Using  such  an 
approach  would  address  the  concern  regarding  courseware  mainte¬ 
nance  cited  by  schools  (see  Appendix  B)  and  reduce  the  cycle  time 
for  maintenance  actions.  This  approach  would  also  help  anticipate  the 
need  for  changes  and  regular  updates  to  courseware,  as  well  as  for  post¬ 
delivery  support,  thus  increasing  overall  IMI  quality,  longevity,  and 
return  on  investment. 

Make  selective  use  of  an  incremental  acquisition  approach.  The 

new  strategy  could  also  benefit  from  the  selective  use  of  an  incremental 
acquisition  approach  for  complex  IMI  or  for  content  areas  that  may 
be  less  well  defined  initially  or  at  risk  for  doctrinal  or  technological 
change  over  the  development  period.  Under  this  approach,  the  over¬ 
all  strategy  could  include  multiple  incremental  deliverables  for  each 
course  (e.g.,  IMI  design  package,  storyboards,  production  of  a  beta  ver¬ 
sion,  planned  revisions).  Such  an  approach  can  reveal  problems  before 
changes  become  difficult  to  implement. 

This  changed  acquisition  strategy  (including  all  its  three  compo¬ 
nents  above)  has  key  benefits  and  promises  greater  overall  effectiveness 
for  Army  DL.  Because  individual  DOs  would  not  need  to  be  com¬ 
peted,  the  administrative  burden  at  ATSC  would  go  down,  and  the 
pressures  of  year-end  funding  would  be  alleviated.  Further,  while  the 
award  process  would  stay  centralized  (e.g.,  the  Northern  Region  Con¬ 
tracting  Center  would  continue  to  serve  as  the  Procuring  Contract¬ 
ing  Office  (PCO)  for  all  TRADOC  requirements  contracts),  contract 
management  and  administration  could,  at  least  selectively,  be  decen¬ 
tralized  to  the  schools  with  regard  to  issues  such  as  the  tailoring  of 
evaluation  criteria  to  schools’  needs  and  the  selection  of  contractors. 
This  option  could  foster  greater  school  involvement  and  buy-in  and 
lead  to  higher-quality  DOs  that  generate  fewer  problems  during  the 
development  period. 
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The  new  arrangements  would  also  foster  continuous  teaming  and 
collaboration  between  schools  and  contractors.  A  central  need  for  pro¬ 
ducing  cost-effective  output  is  continuous  interaction  of  SMEs,  tech¬ 
nical  experts,  and  training  development  experts.  The  establishment  of 
longer-term  partnerships  would  promote  shorter  cycle  times,  increased 
quality,  and  greater  responsiveness  because  the  contractor  would  be 
better  able  to  recruit  an  expert  staff  and  develop  a  better  understanding 
of  the  customer’s  needs. 

Another  benefit  of  the  new  strategic  framework  is  that  mainte¬ 
nance  and  revision  of  existing  content  would  become  a  routine  part  of 
contractors’  work,  significantly  extending  the  life  of  the  content  pro¬ 
duced  and  the  ROI  of  development  efforts.  Through  their  long  associa¬ 
tion  with  particular  schools,  contractors  would  develop  greater  expertise 
in  the  relevant  subject  material  and  would  have  additional  incentives  to 
invest  in  that  expertise.  Finally,  the  lifecycle  focus  and  learning  curve 
benefits  should  lead  to  increased  ROI  from  IMI  investments. 

The  longer-term  initiatives  (as  well  as  the  shorter-term  ones)  would 
initially  cost  more  to  implement  than  the  present  strategy;  however,  the 
efficiency,  quality,  and  ROI  could  greatly  increase  over  time  through 
increased  usage  of  DL  courseware,  shorter  cycle  times,  and  more  timely 
responses  to  the  needs  of  customers  (in  this  case,  the  schools). 

Finally,  the  new  acquisition  strategy  for  IMI  could  better  prepare 
the  Army  to  adopt  new  DF  models  in  the  future,  which  would  allow 
for  more  decentralization  of  IMI  development.  Such  decentralization 
could  be  accomplished  directly  through  the  use  of  local  school  staff,  or 
by  employing  an  on-site  contractor  capability.  More  decentralized  IMI 
production  processes  have  the  capability  to  achieve  further  increases  in 
responsiveness  and  decreases  in  cycle  time  for  IMI.  These  options  and 
their  benefits  are  further  discussed  in  the  next  chapter. 


Ensure  Sufficient  Resources  for  Stand-Alone  IMI 

We  recommend  that  the  Army  increase  its  allocation  of  support 
resources  for  IMI  development  and  support  within  TADFR  While 
TADFP  achieved  a  low  cost  per  hour,  the  overall  return  on  investment 
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was  low  due  to  low  usage,  long  cycle  times,  and  lack  of  responsive¬ 
ness.  The  basic  argument  is  that  while  allocation  of  more  resources  per 
hour  might  reduce  the  number  of  hours  produced,  it  would  promote 
better  ROI  and  also  improve  courseware  quality.  This  action  would 
also  be  needed  to  implement  recommendations  related  to  the  acquisi¬ 
tion  strategy. 

Issues  Regarding  the  Allocation  of  Resources  for  IMI 

Many  of  the  Army’s  IMI  issues  regarding  quality  and  cycle  time  directly 
relate  to  limited  school  support  of  IMI  development.  A  basic  assump¬ 
tion  in  the  Army’s  approach  to  the  development  and  delivery  of  IMI  is 
that  school  support  will  be  available  to  supplement  TADLP  funding. 
When  resources  such  as  GFI/GFM  and  SME  support  are  not  pres¬ 
ent,  the  result  is  an  extended  cycle  time  and  compromises  in  course¬ 
ware  quality.  For  example,  the  DL  process  assumes  that  a  complete 
and  unchanging  set  of  GFI  will  be  available  for  a  relatively  straightfor¬ 
ward  conversion  to  IMI  courseware.  When  the  GFI  is  incomplete,  or 
when  the  proponent  makes  changes  or  additions  during  development 
(as  often  happens)  and  no  more  resources  appear  available  (which  is 
typically  the  case),  the  finished  product  can  be  delayed,  and  originally 
intended  capability  can  be  sacrificed. 

The  models  for  the  production  of  commercial  software  also  sug¬ 
gest  that  the  Army  IMI  model  is  underfunded.  The  Army  model  does 
appear  in  industry  (e.g.,  providing  limited  customer  support)  but  is 
often  associated  with  a  high  risk  of  failure.  Successful  companies  have 
the  kind  of  support  suggested  in  this  section.4 

Our  survey  of  proponent  schools  supports  a  conclusion  that  IMI 
efforts  are  underfunded  for  some  tasks.  Overall,  resourcing  issues  were 
cited  as  a  challenge  by  100  percent  of  the  schools  interviewed  about 
DL,  and  were  identified  as  a  key  issue  by  60  percent  of  the  schools, 
more  than  any  other  obstacle  cited  (see  Appendix  B  for  further  details). 
The  most  commonly  cited  resource  issue  was  “a  lack  of  sufficient  or 
qualified  Subject  Matter  Expert  (SME)  support,”  either  from  the 
school  side,  the  contractor  side,  or  both.  Seventy  percent  of  the  20 


4 


For  elaboration  see  Shanley  et  al.  (2005),  Chapter  3. 
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schools  interviewed  described  this  issue  as  an  obstacle,  and  40  percent 
cited  it  as  a  key  issue.  While  support  for  SMEs  was  commonly  funded 
in  the  award  to  contractors,  according  to  the  schools,  many  contrac¬ 
tors  were  unable  to  find  SMEs  that  were  sufficiently  qualified  to  meet 
school  expectations.  In  addition  to  general  SME  support,  45  percent  of 
the  schools  noted  specific  DL-related  tasks  for  which  TADLP  did  not 
provide  funds,  with  10  percent  identifying  these  unresourced  tasks  as 
a  key  issue.  Respondents  mentioned  the  following  areas  as  other  exam¬ 
ples  of  areas  in  which  resources  were  potentially  lacking: 

•  Completing  up-to-date  GFI. 

•  Specifying  the  learning  objectives  the  IMI  is  to  achieve. 

•  Providing  institutional  support  to  students  (both  technical  and 
substantive)  sufficient  to  ensure  high  participation  in  the  program. 

•  Providing  a  school  capability  for  minor  maintenance  of  course¬ 
ware  or  the  creation  of  simple  DL  products  that  require  rapid 
distribution. 

•  Providing  for  quality  control  processes  and  assessment  of  the  out¬ 
comes  of  DL  training. 

Note  that  the  primary  funding  for  these  functions  comes  from  the 
overall  resources  of  the  proponent  schools  rather  than  from  the  fund¬ 
ing  account  associated  with  TADLP. 

Navy  Success  with  IMI  Supports  the  Need  for  Sufficient  Resources 

Comparison  of  the  Army  program  with  a  Navy  Initiative  also  suggests 
that  the  Army’s  DL  program  could  benefit  from  more  support  in  IMI 
production.  The  Navy  runs  a  DL  program  that,  much  like  TADLP, 
uses  IMI  to  reduce  course  lengths  in  structured  institutional  train¬ 
ing.5  In  contrast  to  the  Army  program,  the  Navy  effort  has  been  highly 
successful  and  has  earned  the  support  of  schools,  units,  and  students. 
Table  4.2  indicates  the  differences  between  the  Army  and  Navy  courses 


5  The  Navy’s  program  differs  from  the  Army’s  in  that  the  target  courses  are  in  the  Navy’s 
A-Schools,  equivalent  to  the  Advanced  Individual  Training  (AIT)  courses  used  in  the  Army 
to  train  entering  soldiers,  and  that  the  IMI  training  takes  place  in  residence  rather  than  from 
a  distance. 
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Table  4.2 

Comparison  of  Army  and  Navy  Resource  Allocation  for  IMI 


Item 

Army 

Navy 

Calculation  of  course 
length  reduction 

30%,  taken  up  front 

10-30%,  based  on 
analysis 

Instructor  resources 

Greatly  reduced 

Held  constant 

Training  development 
resources 

Not  fully  funded 

Fully  funded 

Institutional  support 
to  students 

•  Army  Training  Help  Desk 
(ATHD) 

•  Little  dedicated  time  at  units 

•  Full  support 

•  One-on-one  where 
needed 

Course  testing  and 
evaluation 

•  During  course  development 

•  Some  school  efforts 

•  Pilot  effort  first 

•  Continuous  evaluation 

with  regard  to  resource  support.  The  table  compares  elements  of  the 
Army’s  overall  focus  on  both  resource  and  course  length  reduction 
with  the  Navy’s  focus  on  ensuring  increased  value  to  the  operational 
commander. 

As  indicated  in  the  first  row  of  the  table,  the  Navy  analyzed  the 
content  of  each  course  to  estimate  the  amount  of  time  savings  to  expect 
on  a  case-by-case  basis.  The  results  of  this  analysis  showed  that  the 
course  length  reductions  should  range  from  10  to  30  percent,  depend¬ 
ing  on  the  type  of  training  involved.  In  contrast,  the  Army  avoids 
the  cost  of  those  continuing  analyses  by  using  an  “across-the-board” 
resourcing  formula  that  assumes,  for  that  part  of  the  course  converted 
to  IMI,  course  length  would  drop  by  30  percent  when  residential  hours 
were  converted  to  IMI  hours. 

The  Navy  also  held  the  number  of  instructors  constant,  assign¬ 
ing  one  per  classroom  whether  the  mode  was  traditional  instruction  or 
IMI  instruction  within  the  classroom.  Freed  by  the  IMI  from  platform 
duties,  the  Navy  instructors  worked  to  ensure  course  success  by  men¬ 
toring  students  who  were  falling  behind  the  required  pace  or  failing 
tests. 

The  Navy’s  decision  to  hold  instructor  resources  constant  despite 
the  instructional  capability  of  the  IMI  helped  ensure  that  the  new  and 
shorter  courses  would  provide  equivalent  training  to  the  older  resi¬ 
dential  format  without  undue  student  attrition.  This  strategy  is  in  line 
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with  the  typical  development  cycle  of  technological  improvements  that 
promise  resource  savings.  Research  has  found  that  the  savings  are  real¬ 
ized  not  from  the  introduction  provided  by  technology  itself,  but  from 
the  reengineering  of  existing  processes  to  capitalize  on  the  technology’s 
strength.  Thus,  success  tends  to  come  in  two  phases,  the  first  focusing 
on  making  the  new  technology  work  at  least  as  well  as  the  old  technol¬ 
ogy,  and  the  second  focusing  on  identifying  possible  resource  savings.6 
In  the  use  of  IMI  in  the  training  context,  this  means  focusing  first 
on  achieving  equivalent  training  outcomes,  then  on  reducing  course 
length  and  other  savings.  The  Army’s  approach  has  been  to  reduce  the 
process  to  a  single  step,  assuming  that  all  resources  could  be  saved  up 
front  while  implementing  a  new  training  approach  with  undetermined 
effects  on  quality. 

Table  4.2  shows  additional  resources  the  Navy  chose  to  invest 
in  quality  in  order  to  add  to  the  value  of  the  outcome.  These  include 
training  development,  student  support,  and  course  assessment. 

Recommendations  for  Improvement 

Our  assessment  leads  us  to  conclude  that  the  Army’s  IMI  approach  is 
not  sufficiently  resourced  on  a  “per  hour”  basis.  Thus,  we  recommend 
that  the  Army  allocate  more  resources  to  each  training  hour  it  decides 
to  fund.  A  contractual  means  for  supplying  these  resources  has  been 
discussed  in  the  previous  section,  under  recommendations  for  changes 
in  the  Army’s  acquisition  strategy  for  courseware. 

Provide  for  additional  resource  categories.  The  following 
resource  categories  may  need  to  be  newly  funded  or  to  have  an  increase 
in  funding: 

•  SME  support. 

•  GFI  production. 

•  Front-end  analysis. 

•  Instructor  support  to  students  during  delivery. 


6  For  example,  in  the  Navy’s  case  it  may  turn  out  that  once  new  processes  have  evolved, 
one  instructor  can  be  shown  to  successfully  mentor  a  greater  number  of  students  than  the 
number  in  a  typical  residential  class. 
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•  Improvements  and  revisions. 

•  Assessments. 

Some  of  these  elements  (e.g.,  need  for  SME  support)  were  identi¬ 
fied  by  the  schools  as  obstacles  to  their  present  IMI  efforts.  In  at  least 
one  case  (instructor  support  during  delivery),  TRADOC  has  already 
acknowledged  the  need.  A  pilot  program  involving  the  Armor  School 
has  led  to  the  assignment  of  reserve  component  instructors  to  provide 
added  support  to  students  during  the  DL  phase  of  the  course. 

Increases  in  funding  for  the  elements  listed  above  would  appear 
to  make  sense  given  current  Army  constraints.  For  example,  SME  sup¬ 
port  and  resources  for  the  production  of  improved  GFI,  at  least  in  some 
select  cases,  would  understandably  be  required  because  schools  report 
that  TRADOC  does  not  have  a  training  development  capability  suffi¬ 
cient  to  support  the  production  of  IMI.  Estimates  of  how  much  would 
be  needed  can  be  determined  through  pilot  efforts  as  part  of  an  R&D 
effort. 

Note  that  this  recommendation  means  bringing  more  of  the  sup¬ 
port  needed  for  IMI  courses  within  the  Management  Decision  Evalu¬ 
ation  Package  (MDEP)7  associated  with  TADLP,  rather  than  leaving  it 
as  an  unfunded  requirement  for  the  individual  schools.  Moving  more 
of  the  funding  under  the  TADLP  MDEP  would  help  “fence”  funds 
needed  for  DL  support.  For  example,  to  further  address  the  need  for 
sufficient  SMEs  to  assist  with  training  development,  the  TADLP  could 
generate  separate  non-personnel  services  contracts  funded  out  of  the 
TADLP  MDEP  and  overseen  by  the  schools.  This  approach  would  give 
schools  more  direct  control  over  the  use  of  expert  manpower  and  help 
alleviate  problems  of  underqualified  SMEs  performing  training  devel¬ 
opment  tasks. 

Since  total  resources  for  the  development  of  DL  courseware  are 
not  likely  to  increase,  the  additional  investments  proposed  would  mean 
that  fewer  courses  will  be  funded,  at  least  in  the  short  run.  However, 


7  MDEPs  divide  Army  programs  into  subcomponents  for  the  purposes  of  managing 
resources  in  the  budgeting  process.  As  a  supporting  capability  for  training,  TADLP  has  a 
designated  MDEP  through  which  it  receives  much  of  its  funding. 
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the  number  need  not  be  large,  as  additional  resources  are  anticipated 
to  represent  only  a  small  percentage  of  total  cost.  In  addition,  over  the 
longer  term,  these  investments  (properly  made)  could  well  result  in  a 
higher  payback  for  the  program.  This  could  be  effected,  for  example, 
by  increasing  the  average  lifecycle  of  content,  the  usage  of  available 
products,  and  the  course  completion  rates. 

It  is  important  to  note  that  the  Army  is  not  necessarily  “stuck” 
with  higher  costs  per  IMI  hour  and  lower  amounts  of  IMI  courseware. 
By  broadening  its  current  approach  to  include  both  DL  development 
and  execution,  the  Army  could  bring  other  costs  down  to  balance  out 
the  short-term  cost  increases  while  increasing  the  number  of  students 
trained  via  DL.  For  example,  by  the  end  of  the  assessment  period,  the 
Army  is  beginning  to  pursue  approaches  that  couple  blended  learning 
with  instructor  development  of  appropriate  types  of  IMI  courseware. 
These  approaches  may  well  cost  less  than  the  investments  just  discussed 
and  lead  to  an  actual  increase  in  the  Army’s  ability  to  conduct  DL 
instruction  even  with  current  TADLP  budgets.  These  options  are  dis¬ 
cussed  in  the  next  chapter  of  this  report. 


Undertake  More  Systematic  Process  Improvements  to 
Further  Reduce  Cycle  Times 

In  Chapter  Two  we  reviewed  the  process  for  IMI  production  used  by 
TADLP.  As  shown  in  Chapter  Three,  TRADOC  schools  require  sig¬ 
nificantly  more  time  than  does  the  commercial  sector  to  carry  out  all 
the  steps  required  to  create  and  field  new  IMI.  This  remains  the  case 
even  after  accounting  for  the  Army’s  recent  gains  in  this  area.  There 
appears  to  be  an  opportunity  for  the  Army  to  further  reduce  its  cycle 
time.  We  recommend  that  the  Army  institute  a  more  systematic  pro¬ 
cess  improvement  effort  that  will  allow  for  both  near-term  and  ongoing 
reductions  in  cycle  time.  We  also  recommend  some  specific  changes  in 
the  Army’s  practices  in  the  near  term.  To  the  extent  that  the  acquisi¬ 
tion  strategy  and  process  is  redesigned  in  the  future  and  supporting 
resources  increased  (e.g.,  along  the  lines  discussed  in  the  prior  sections), 
additional  opportunities  for  streamlining  processes  could  also  arise. 
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Issues  Regarding  Cycle  Time 

There  are  a  number  of  known  issues  that  appear  to  contribute  to  the 
long  development  times  that  have  been  identified  earlier.  We  learned 
of  many  of  these  issues  through  our  review  of  the  IMI  acquisition  and 
development  process  and  through  discussions  with  TD  representatives 
from  the  schools  and  with  contractors: 

•  Nominations  for  DL  conversions  are  developed  through  an  annual 
cycle  during  which  a  single  “batch”  of  proposals  is  submitted  for 
a  single  set  of  approvals,  which  are  funded  at  the  end  of  the  fiscal 
year.  Conducting  the  process  in  this  way  adds  up  to  a  year  to  the 
development  time. 

•  The  development  process  has  too  many  steps  and  has  involved 
frequent  changes  in  policies/practices. 

•  The  process  involves  many  stakeholders  and  organizations,  and 
many  documents  and  signoffs. 

•  The  process  is  bureaucratic  and  inflexible  to  change  or  streamlin¬ 
ing.  For  example,  the  same  one-year  contract  is  used  for  small  and 
large  IMI  product-development  efforts.  The  process  lacks  “some¬ 
one  in  charge,”  i.e.,  a  specific  person  or  office  with  overall  author¬ 
ity  and  responsibility  in  each  school,  to  manage  development  and 
provide  continuous  feedback/guidance  to  the  contractor  to  keep 
development  moving. 

•  Schools  lack  resources  to  develop  high-quality  GFI/GFM,  pro¬ 
vide  SME  support  during  IMI  development,  or  carry  out  timely 
reviews  of  content  from  contractors.  The  need  for  changes  often 
leads  to  no-cost  time  extensions  or  reduced  quality. 

•  Testing  for  SCORM  and  ALMS  historically  has  added  substan¬ 
tial  delays. 

•  Schools  and  contractors  have  inadequate  time  for  the  kinds  of 
negotiation  and  collaboration  that  could  well  bring  about  faster 
development  times. 

•  Best  practices  are  not  shared  between  schools  or  contractors. 

Such  issues  are  common  in  legacy  processes  that  have  evolved 
over  long  periods  and  have  spanned  different  technologies,  organiza- 
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tional  structures/changes,  contracting  approaches,  and  management 
philosophies. 

TRADOC  has  been  aware  of  many  of  these  issues  and  has  imple¬ 
mented  a  number  of  initiatives  to  reduce  cycle  time,  including  incre¬ 
mentally  testing  lessons  for  SCORM/ALMS  compliance  in  advance  of 
the  completion  of  the  entire  course,  expediting  the  post- award  meeting, 
reducing  the  requirements  for  the  instructional  media  design  package 
(IMDP),  and  conducting  a  Lean  Six  Sigma  study  on  streamlining  the 
courseware  testing  process.  More  recently,  TRADOC  has  initiated  a 
“tiger  team”  approach  to  understanding  and  improving  the  speed  of 
the  process.  These  efforts  appear  to  have  led  to  some  recent  decreases  in 
IMI  product-development  times,  but  the  data  do  not  exist  to  let  man¬ 
agers  know  which  of  these  efforts  were  the  most  successful  or  to  guide 
next  steps.  Having  such  information  could  well  be  crucial  to  the  design 
of  future  initiatives,  as  ongoing  improvements  in  cycle  time  will  likely 
require  detailed  knowledge  about  the  source  of  delays  in  the  process. 

Recommendations  for  Improvement 

Our  assessment  leads  us  to  the  following  recommendations  for  reduc¬ 
ing  cycle  time. 

Expand  the  capability  for  more  systematic  process  improve¬ 
ments.  Whether  using  a  new  or  existing  acquisition  strategy,  the  Army 
should  expand  its  efforts  to  engage  in  more-structured  and  continuous 
process  improvement  to  help  identify  and  eliminate  or  speed  up  the 
processes  responsible  for  delays  in  IMI  development.  Although  there 
are  many  methods  for  implementing  continuous  improvement,  such  as 
Six  Sigma  or  Total  Quality  Management  (TQM),8  their  common  basic 
underlying  steps — “define,”  “measure,”  and  “improve”  processes — can 
also  be  applied  to  IMI  development.9 


8  Six  Sigma  is  a  business  management  strategy  originated  by  Motorola,  USA  that  seeks 
to  improve  the  operational  performance  of  an  organization  by  eliminating  variability  and 
waste.  TQM  is  a  strategic  approach  to  management  aimed  at  embedding  awareness  of  qual¬ 
ity  in  all  organizational  processes. 

9  For  a  number  of  years,  RAND  has  been  involved  in  developing  and  implementing 
very  successful  process  improvement  activities  with  U.S.  Army  logistics  organizations  (see 
Dumond  et  ah,  2001).  The  same  basic  processes  can  apply  to  IMI  development,  although 
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Case  studies  of  process  improvement  efforts  that  have  succeeded 
and  failed  have  yielded  a  short  list  of  required  components  for  success. 
The  effort  must  have: 

•  A  “champion,”  a  senior  leader  who  “owns”  the  process  at  some 
level  and  will  commit  oversight  and  resources  to  the  hard  work  of 
process  improvement. 

•  A  cross-functional,  empowered  “process  improvement  team”  to 
guide  specific  improvement  efforts;  members  must  include  the 
major  stakeholder  groups  in  the  process. 

•  Time  and  travel  resources  to  carry  out  process  improvement 
methods,  beginning  with  “process  walks”  to  define  the  process 
in  detail. 

•  Strategic  and  tactical  data  collection  of  process  performance  mea¬ 
sures  into  databases  that  can  be  used  to  assess  the  success  or  fail¬ 
ure  of  improvement  efforts. 

•  Continuous  monitoring  and  improvement. 

Table  4.3  shows  the  main  steps  in  continuous  process  improvement. 

Once  the  basic  steps  in  process  improvement  have  been  taken,  the 
team  should  assess  whether  the  potential  improvement  had  the  desired 
effect.  If  the  improvement  has  worked,  it  will  be  evident  in  reduced 
process  times.  The  cycle  of  process  improvement  is  then  repeated,  as 
additional  improvements  are  implemented. 

The  RAND  Arroyo  Center  team,  with  the  support  of  personnel 
and  data  from  ATSC,  began  the  process  of  defining  the  IMI  develop¬ 
ment  process  and  identifying  the  potential  causes  of  the  process  delays 
described  earlier.  This  initial  investigation  started  by  defining  the  steps 
in  the  process  at  the  highest  level  and  then  mapping  out  the  steps  in 
more  detail  for  critical  segments  of  the  process.  This  mapping  was  done 
both  for  the  Army  and  for  a  generic  commercial  IMI  development  pro- 


there  are  important  differences  in  the  number  of  iterations  possible  for  improving  logistics 
work  (processes  function  daily  and  generate  large  amounts  of  performance  data)  compared 
to  IMI  development  (currently  annual  cycle,  few  products). 
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Table  4.3 

Major  Steps  in  Continuous  Process  Improvement 


Define  the  Process 

•  Lay  out  the  process  in  detail,  from  "end  to  end,"  including  all  stakeholders, 
documents,  decisions,  and  methods  used  to  move  information. 

•  Complete  a  "process  walk"  in  which  improvement  team  members  "pin  an  order 
to  their  collar"  and  follow  the  entire  flow  of  the  process. 

•  Combine  process  steps  into  "segments"  which  should,  if  possible,  be  linked  to 
individual  organizations  that  "own"  the  segment. 

•  Establish  metrics  based  on  discrete  time  stamping. 

Measure  the  Performance  of  the  Process 

•  Use  previously  determined  metrics  to  track  performance  of  the  process, 
including  historical  performance,  if  possible. 

•  Track  the  segment  times  in  real  time  and  make  the  results  available  to  the 
process  improvement  team  and  other  participants  so  they  can  provide  feedback. 

•  Develop  a  baseline  of  performance  against  which  to  apply  improvement  efforts. 

Improve  the  Process 

•  Propose  changes  in  the  process  to  reduce  cycle  time. 

•  Identify  "low-hanging  fruit"  in  the  process,  i.e.,  easy-to-implement  changes 
that  produce  quick  improvements,  including 

-  Eliminating  unnecessary,  non-value-adding  steps. 

-  Speeding  up  necessary  steps. 

-  Automating  or  eliminating  explicit  handoffs  and  approvals. 

•  Communicate  improvements  to  segment  owners  and  implementers,  and 
establish  ownership  of  the  improvement  effort. 

•  Provide  performance  data  back  to  team  and  all  stakeholders. 


cess.10  We  based  our  mapping  of  the  Army’s  IMI  process  on  guidance 
in  TRADOC  Pamphlet  350-70-12,  information  and  data  from  inter¬ 
views  with  Army  civilians  at  ATSC  and  interviews  with  training  devel¬ 
opers  and  instructors  at  the  a  number  of  the  Army’s  proponent  schools. 

We  divided  the  process  into  two  major  segments  based  on  the 
date  of  “signing  a  contract.”  Contract  signing  marks  the  end  of  one  set 
of  activities  (planning  and  preparation)  and  the  beginning  of  another 


10 


Based  on  interviews  with  commercial  IMI  developers  and  trainers. 
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(development).  The  planning  and  preparation  segment  encompasses  all 
the  activities  from  the  time  the  school  first  perceives  the  need  for  an 
IMI  course  until  the  time  when  the  contract  is  signed  to  develop  the 
course.  The  development  segment  covers  all  activities  from  the  signing 
of  the  contract  until  the  point  when  actual  Army  learners  are  using  the 
IMI  content.  The  two  major  segments  have  been  broken  down  fur¬ 
ther  into  many  underlying  steps,  which  involve  many  supporting  docu¬ 
ments  and  stakeholders. 

Although  many  stakeholders  might  interact  in  the  steps  involved 
in  a  major  process  segment,  the  break  points  should  be  designed  so  that 
one  main  stakeholder  is  primarily  accountable  for  the  performance  of 
all  the  steps  in  the  segment.  For  example,  for  the  period  from  contract 
signature  and  work  commencement  to  Army  learners  actually  using 
courseware,  the  segments  and  owners  might  be: 

•  “Contract  to  First  Lesson”:  Main  segment  owner  is  the  propo¬ 
nent  school  overseeing  the  IMI  development  and  interacting 
closely  with  the  contractor. 

•  “First  Lesson  to  Final  Lesson  Delivered”:  Main  segment  owner  is 
the  proponent  school  overseeing  the  IMI  development  and  inter¬ 
acting  closely  with  the  contractor. 

•  “Final  Lesson  to  Live”:  Main  segment  owner  is  ATSC,  which 
performs  testing  and  implementation  steps  to  get  the  content  cer¬ 
tified  and  running  on  Army  servers. 

•  “Live  to  In  Use  by  Learners”:  Main  segment  owner  is  the  propo¬ 
nent  school,  which  controls  when  and  how  the  content  is  accessed 
by  learners. 

Make  near-term  improvements.  Based  on  the  interviews  and 
data  that  were  assembled  to  create  the  process  map,  we  documented 
a  number  of  important  issues  underlying  the  slow  IMI  development 
times.  We  also  developed  examples  of  a  number  of  possible  solutions 
to  the  issues  via  discussions  with  personnel  from  ATSC,  staff  at  pro¬ 
ponent  schools,  contractors,  and  commercial  subject  matter  experts. 
These  are  not  meant  to  be  exhaustive,  but  instead  represent  the  kinds 
of  improvement  actions  that  might  be  candidates  for  process  improve- 
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ment  efforts.  We  included  in  our  list  what  we  saw  as  “low-hanging 
fruit”:  actions  that  can  be  taken  to  address  fairly  obvious  aspects  of  the 
process  to  produce  significant  changes  early  on. 

Table  4.4  presents  possible  near-term  improvement  actions. 

The  simplest  and  most  direct  way  to  reduce  end-to-end  cycle 
time  is  to  release  IMI  funds  at  the  beginning  of  the  year.  The  prac¬ 
tice  of  not  releasing  DA-provided  IMI  development  funds  until  near 
the  end  of  the  year  increases  cycle  time  substantially  by  delaying  the 
start  of  needed  courseware  development.  It  also  decreases  the  effective¬ 
ness  of  contractor  selection  and  school-contractor  interaction.  While 
leading  to  the  most  immediate  improvement,  this  option  does  require 
TRADOC  command  support. 


Table  4.4 

Recommended  Near-Term  Actions  to  Reduce  IMI  Process  Development  Time 


Issues 


Potential  Solutions 


Annual  single  "batch"  cycle  of  a  Release  IMI  funds  at  the  beginning  of  the 
proposals  slows  process,  as  does  year  and  initiate  quarterly  submissions  and 
the  practice  of  year-end  funding  approvals  of  projects 


Process  complexity  leads  to  Continue  to  refine  application  of  process 

delays:  too  many  steps,  documents,  improvement  methods  from  "end  to  end," 
signoffs,  changes  ensuring  all  elements  of  success  are  in  place 

Uniform  length  contract  Tailor  length  to  size  of  work  effort  for  specific 

inappropriate  for  all  size  contracts  products 


Funding  surge  for  DL  contracts  at  Work  with  TRADOC  to  help  leadership 
end  of  FY  understand  effects  of  funding  strategy  and 

change  to  distribute  funds  throughout  year 


Testing  for  full  ALMS  and  SCORM  Evaluate  underlying  causes  of  delays,  review 
compliance  adds  to  fielding  delays  value  of  "blanket"  requirement  for  all  aspects 

of  SCORM  conformance,  consider  waivers  for 
low  reuse  content 


Best  practices  for  speeding 
development  are  not  widespread 
in  the  areas  of  local  development 
and  contractor  relationship 
management  practices 


Create  incentives  and  opportunities  to  share 
best  practices  via  face-to-face  and  virtual 
meetings,  e.g., 

•  sessions  at  the  spring  DL  meeting 

•  improved  web-based  community  of 
Army  DL  developers  and  contractors 
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Also  of  key  importance  among  the  suggestions  in  Table  4.4  is  the 
recommendation  to  continue  to  implement  and  refine  the  application 
of  process  improvement  methods,  and  to  make  them  an  ongoing  part 
of  TRADOC’s  approach  to  IMI  development. 

Begin  improvements  with  longer-term  payoffs.  Longer-term 
solutions  to  address  known  issues  that  lengthen  the  IMI  development 
cycle  are  listed  in  Table  4.5.  Note  that  two  of  these  solutions,  changing 
the  acquisition  strategy  and  providing  TADLP  funds  for  assessment, 
are  discussed  earlier  in  this  chapter. 

Implementing  the  types  of  solutions  to  speed  IMI  development 
listed  in  Tables  4.4  and  4.5  is  made  easier  in  the  context  of  what 
Hammer  calls  a  “business  process  culture.”11  Such  an  organizational 
culture  would  mean  that  business  units  embrace  a  cross-functional 
approach  to  business  processes.  At  the  same  time,  all  aspects  of  the 


Table  4.5 

Recommended  Longer-Term  Actions  to  Reduce  IMI  Process  Development 
Time 


Issues 

Potential  Solutions 

Design,  development,  review, 
and  revision  of  IMI  content 
require  a  long  time  to  carry  out 

Experiment  with  rapid  development  methods 
that  use  short,  very  focused  sessions  with 

SMEs  and  developers;  use  rapid  prototyping 
methods  to  test  user  interfaces  and  content; 
use  web-based  collaboration  tools  for 
content  reviews 

Lack  of  resources  at  schools  to 
provide  appropriate  project 
management,  contractor 
relationship  management, 

SME  expertise 

Provide  funding  to  schools  to  increase 
expertise  to  oversee  and  speed  IMI  product 
development 

Inflexible  contract  type  that 
limits  interaction,  tailoring  of 
content  in  timely  manner 

Change  acquisition  strategy:  Move  from 
"firm,  fixed-price"  to  "requirements" 
contract  type  for  software  acquisition, 
maintenance 

"Real"  process  improvement 
requires  continuous  cycles 

Institutionalize  practices,  data  collection/ 
sharing,  culture  throughout  TRADOC  DL 
community 

n 


Hammer  and  Champy  (1993). 
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organization  also  take  a  “customer-oriented”  view  of  how  to  provide 
the  best  service.  For  example,  proponent  schools  are  “customers”  for 
TRADOC  services,  “Army  learners”  are  the  customers/consumers  for 
IMI  content,  and  unit  commanders  are  the  customers  for  trained  sol¬ 
diers.  The  intellectual  foundation  supporting  the  “business  process  cul¬ 
ture”  is  the  perspective  of  “process”  or  “systems  thinking,”12  in  which 
the  organization  and  customers  are  viewed  as  an  interrelated  set  of  enti¬ 
ties  with  complex  feedback  and  feed-forward  loops. 


Increase  Local  Participation  in  IMI  Production  and 
Contract  Administration 

To  achieve  multiple  improvements  in  program  outcomes,  we  recom¬ 
mend  two  ways  that  TADLP  can  increase  proponent  participation  in 
IMI  production.  First,  TADLP  should  increase  the  practice  of  produc¬ 
ing  some  IMI  in  house.  Proper  selection  of  content  for  local  produc¬ 
tion  of  IMI  can  reduce  variable  costs  and  cycle  time  and  increase  the 
responsiveness  of  IMI  to  the  need  for  change.  Second,  in  cases  where 
IMI  production  is  still  contracted  out,  TADLP  should  decentralize 
selected  aspects  of  contract  management  and  administration  in  order 
to  increase  responsiveness  and  the  quality  of  the  IMI  product. 

Issues  Regarding  Local  Participation  in  IMI  Development 

Early  in  the  history  of  TADLP,  IMI  production  was  largely  decen¬ 
tralized,  with  proponent  schools  producing  nearly  all  the  content  in 
house.  However,  many  believed  this  level  of  decentralization  often  led 
to  unacceptable  outcomes  and  practices,  including  the  use  of  variable 
and  often  inefficient  methods  of  production,  and  products  of  unaccept¬ 
able  quality.  These  outcomes  were,  in  part,  responsible  for  the  transi¬ 
tion  of  TADLP  to  a  contractor-only  production  model,  using  a  central¬ 
ized  contracting  vehicle. 

As  shown  by  the  assessment  in  Chapter  Three,  the  contractor- 
only  model  of  IMI  production  also  does  not  fare  well  with  respect 


12 


Senge  (1994). 
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to  the  DL  measures  of  effectiveness  identified  in  this  report.  Further, 
emerging  technologies  for  IMI  production  have  changed  the  dynamics 
involved  in  using  in-house  versus  contracted  staff. 

In  particular,  a  key  enabling  factor  in  the  in-house  approach  is 
the  greatly  increased  capability  of  authoring  tools  to  produce  and  allow 
self-maintenance  of  web-based  content.13  These  tools  allow  instruc¬ 
tors  to  become  developers  of  DL  content  without  being  web  program¬ 
mers.14  Content  can  be  up  to  interactivity  Level  2  and  occasionally 
even  Level  3.  This  is  accomplished  by  providing  rich  templates  for  navi¬ 
gation  of  web-based  content,  “drag  and  drop”  programming  of  IMI 
features  from  libraries  of  generic  objects  (e.g.,  buttons),  and  “record¬ 
ing”  of  voice  and  interaction  with  software  via  capturing  of  all  interac¬ 
tions  on  computer  screens.  Earlier  efforts  by  TRADOC  to  standardize 
web-based  authoring  tools  via  “Toolbook”  occurred  when  such  tools 
were  still  early  in  their  evolution  and  required  significant  programming 
skills.  The  newer  tools,  including  Adobe  “Captivate”  and  “Camtasia,” 
allow  SMEs  with  basic-level  tool  expertise  to  author  Level  1-2  IMI.15 

Recommendations  for  Improvement 

In  light  of  the  greatly  increased  capability  of  authoring  tools  to  produce 
and  maintain  web-based  content,  we  recommend  that  TADLP  expand 
the  concept  of  in-house  production.  Recent  Army  efforts  to  bring  more 
development  in  house  (described  in  full  in  the  following  chapter)  show 
that  organic  development  can  be  achieved  at  only  a  fraction  of  the  cost 


13  Brandon  Hall  Research  Staff  (2006). 

14  It  is  important  to  note  that  the  evolution  of  powerful  authoring  tools  is  independent  of 
the  possible  breadth  of  IMI  instructional  quality  that  is  produced  with  those  tools.  High- 
quality  IMI  requires  good  instructional  design  and  content,  as  well  as  effective  delivery. 
Quality  assurance  will  always  be  an  issue. 

15  A  similar  phenomenon  occurs  for  higher  levels  of  interactivity  (Levels  3  and  4)  for  some 
“soft  skill”  content  (e.g.,  engaging  in  role-playing  or  problem  solving).  For  example,  basic 
simulations  that  use  video  vignettes  with  branching  require  only  basic  videography  and 
simple  video  editing  skills  that  are  relatively  easy  to  acquire.  Achieving  higher  levels  of 
interactivity  for  “hard  skills”  (e.g.,  simulating  physical  systems  or  creating  virtual  worlds 
where  special  relationships  are  important)  continues  to  require  greater  levels  of  programming 
expertise. 
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of  the  Army’s  contracted  DL  rate  and,  equally  important,  in  only  a 
fraction  of  the  time  needed  to  develop  comparable  contracted  courses. 
Further,  in-house  production  is  likely  to  lead  to  cost  and  time  sav¬ 
ings  over  the  course  lifecycle  because  maintenance  and  updating  of  the 
course  can  be  completed  locally  Perhaps  most  important,  increasing 
in-house  production  allows  local  schools  to  be  responsive  to  the  ongo¬ 
ing  need  for  courseware  change,  at  a  level  that  is  comparable  to  that 
achieved  with  residential  POIs. 

Selected  schools  have  already  been  resourced  (outside  of  TADLP) 
for  local  production,  and  this  practice  should  be  substantially  expanded. 
The  benefits  of  organic  development,  however,  can  be  accessible  to  the 
Army  more  broadly  only  with  the  provision  of  new  processes  and  kinds 
of  support,  including  sufficient  manpower  resources  at  the  local  level.16 
In  the  near  term,  embedding  contractors  might  be  more  feasible  than 
acquiring  new  Army  civilians.  As  the  number  of  projects  increases, 
TRADOC  may  find  it  more  efficient  to  turn  increasingly  toward  DA 
civilian  positions  as  the  best  way  to  develop  most  IMI  content. 

Even  as  schools  begin  producing  their  own  IMI,  DL  contractors 
will  still  be  required  in  many  contexts.  It  is  anticipated  that  in-house 
IMI  production  would  be  combined  with  contracting,  in  some  cases  to 
assist  schools  with  some  aspects  of  local  production  (e.g.,  complete  the 
IMI  with  higher  levels  of  interactivity),  and  in  other  cases  to  continue 
to  produce  entire  modules  (e.g.,  when  the  school  has  little  experience 
with  IMI  or  only  a  small  amount  of  content  to  convert). 

The  acquisition  strategy  described  in  an  earlier  recommendation 
provided  in  this  chapter  can  serve  as  an  enabler  for  building  in  more 
local  capability  in  the  schoolhouse.  The  requirements  contract  is  flex¬ 
ible  enough  to  serve  the  Army  during  the  transition  toward  more  local 
production  of  IMI.  For  example,  it  would  be  straightforward  to  define 


16  Over  time,  this  expansion  will  make  it  necessary  to  address  some  deficiencies  related  to 
training  development  capability  that  have  emerged  in  the  schools  after  years  of  capability 
cuts.  For  example,  TRADOC  will  need  to  invest  in  the  revitalization  of  curriculum  develop¬ 
ment  within  schools,  including  those  aspects  related  to  learning  analysis  and  course  design, 
and  will  need  to  provide  additional  support  for  task  selection  with  regard  to  the  many  evolv¬ 
ing  approaches  to  DL  (these  will  be  reviewed  in  Chapter  Five). 
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varying  scopes  of  work  in  requirements  contracts  based  on  individual 
schools’  needs  for  contracting  capability. 

To  maintain  the  integrity  of  TADLP  contracting  as  a  whole,  the 
contracting  process  would  still  have  centralized  elements,  even  as  the 
schools  became  more  involved.  For  example,  the  Northern  Region 
Contracting  Center  (NRCC)  would  continue  to  serve  as  the  Procur¬ 
ing  Contracting  Office  (PCO)  for  all  TRADOC  requirements  con¬ 
tracts.  Further,  NRCC  would  establish  common  solicitation  docu¬ 
ments  and  uniform  statements  of  work  for  all  requirements  contracts. 
However,  contract  management  and  administration  could  be  appro¬ 
priately  decentralized  in  ways  that  facilitate  more  local  participation 
in  the  IMI  production  process  and  higher-quality  IMI  content.  The 
following  changes  are  recommended. 

Evaluation  criteria.  Although  the  NRCC  would  establish  a 
common  framework  for  proposal  evaluation  (including  draft  evalua¬ 
tion  criteria),  the  proponent  and  local  contracting  office  would  have  the 
ability  to  tailor  technical  evaluation  criteria  to  make  them  specific  to 
local  needs.  Technical  criteria  should  emphasize  the  vendor’s  demon¬ 
strated  expertise  in  the  areas  specified  by  the  local  contract  and  in  the 
specific  held  of  content.  This  would  include  the  contractor’s  ability  to 
keep  SMEs  on  hand  to  help  address  the  courseware  content.  It  could 
also  include  evaluation  of  past  performance  on  other  contracts  in  such 
areas  as  quality,  timeliness,  and  cost  control. 

Selection  of  contractors.  In  evaluating  proposals,  the  local  con¬ 
tracting  office  and  proponent  can  also  serve  as  the  technical  evalua¬ 
tion  board,  evaluating  proposals  in  accordance  with  the  criteria  set  by 
NRCC.  This  will  establish  local  ownership  of  the  contractor  selection 
decision.  The  results  of  the  technical  evaluation  board  would  be  for¬ 
warded  to  NRCC  as  the  source  selection  decision  authority.  NRCC,  in 
concert  with  ATSC,  would  review  the  board’s  recommendation,  select 
the  contractor,  and  award  the  requirements  contract. 

Administration.  Although  NRCC  would  remain  the  procuring 
contracting  office,  administration  of  the  contract  could  also  be  del¬ 
egated  to  the  local  contracting  office.  Medical  courseware,  for  example, 
could  be  administered  by  the  contracting  office  at  Fort  Sam  Houston. 
Army  contracting  officer  duties  could  be  fully  delegated  for  each  con- 


86  Making  Improvements  to  The  Army  Distributed  Learning  Program 


tract  to  the  appropriate  local  office.  Alternatively,  to  retain  configura¬ 
tion  control,  NRCC  could  grant  the  local  office  ordering  authority  for 
the  local  requirements  contract,  with  the  ability  to  generate  work  state¬ 
ments  with  the  local  school  and  issue  the  delivery  orders. 


Strengthen  the  IMI  Quality  Evaluation  Component 

Our  fourth  recommendation  is  that  the  Army  strengthen  program- 
level  evaluation  of  IMI  content  produced  by  the  schools.  While  the 
evaluation  component  for  DL  courseware  has  been  delegated  to  the 
schools  up  to  this  point,  we  argue  that  instituting  program-level  qual¬ 
ity  evaluation  would  have  multiple  benefits  and  could  be  implemented 
cost-effectively.17 

Issues  Regarding  the  Evaluation  of  IMI  Quality 

Our  baseline  assessment  concluded  that  there  is  no  effective  effort  to 
assess  course  quality  at  the  program  level.  Although  several  types  of 
independent  training  evaluation  occur  at  the  proponent  school  level, 
TADLP  does  not  have  sufficient  data  and  data  integration  to  gauge 
overall  program  effectiveness  (see  Chapter  Three). 

Improved  program-level  assessment  of  IMI  content  may  well  be 
necessary  to  ensure  content  quality  and  IMI  effectiveness  in  training. 
Our  baseline  assessment  suggested  a  number  of  significant  deficiencies 
in  TADLP  courseware  produced  to  date  (see  Chapter  Three).  In  par¬ 
ticular,  our  evaluation  of  a  sample  of  courses  suggested  that  IMI  con¬ 
tent  was  somewhat  “thin”  in  some  areas,  as  exemplified  by  pedagogical 
shortcomings  and  the  lower-than-required  IMI  “levels  of  interactivity” 
for  much  of  the  content.  Such  deficiencies  may  also  have  additional 
indirect  effects  on  TADLP;  for  example,  poor-quality  courses  could 
contribute  to  low  usage  rates  of  IMI  modules. 

Our  pilot  assessment  might  be  considered  preliminary  in  a  number 
of  ways.  For  example,  we  were  able  to  assess  only  a  small  sample  of 
courses  produced  in  TADLP.  Most  of  the  courses  were  provided  on 


17  Straus  et  al.  (2009)  is  a  more  detailed  treatment  of  this  subject. 
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CD,  which  meant  that  we  did  not  have  access  to  fully  functioning 
courseware  or  course  tests.  Furthermore,  courseware  being  developed 
may  not  have  the  same  deficiencies  as  the  courseware  reviewed  in  our 
analysis.  Since  the  courseware  production  cycle  is  relatively  long  and 
our  analysis  reviewed  only  fielded  courses,  some  of  the  courses  in  our 
sample  were  funded,  and  work  begun,  in  FY  2005  or  before.  ATSC 
has  continually  added  improvements  to  TADLP  processes  since  that 
time.18 

However,  there  are  other  indicators  that  the  quality  of  DL  content 
needs  additional  focus.  Our  study  of  the  courseware  development  pro¬ 
cess  found  that  originally  intended  content  capability  was  sometimes 
sacrificed  in  order  to  accommodate  other  needed  changes  in  the  work 
without  increasing  the  value  of  the  contract.  As  we  saw  in  the  previous 
section,  schools  felt  that  insufficient  resources  were  allocated  to  ensur¬ 
ing  content  effectiveness.  More  directly,  our  interviews  with  training 
development  staff  from  proponent  schools  pointed  to  significant  issues 
with  the  quality  of  the  content  produced  under  TADLP.  In  particular, 
over  a  third  of  the  schools  indicated  they  were  “somewhat  dissatisfied” 
or  “very  dissatisfied”  with  the  DL  content  produced  under  TADLP.19 
Those  dissatisfied  with  quality  frequently  commented  that  they  saw 
inadequate  quality-control  processes  within  TADLP. 

Program-level  assessment  of  courseware  quality  is  critical  for  stra¬ 
tegic  management  for  TADLP  and  will  help  in  gaining  support  and 
resources  from  key  stakeholders.  A  program-level  assessment  of  IMI 
quality  could  aid  in  strategic  planning  and  management:  to  under¬ 
stand  TADLP  outputs,  to  manage  budgets  devoted  to  increasing  qual¬ 
ity,  and  to  identify  and  implement  needed  improvements  to  processes 
that  affect  quality.  Moreover,  ensuring  and  documenting  the  quality 
of  IMI  courseware  is  especially  important  to  show  the  value  of  this 


18  For  example,  IMI  levels  have  been  better  defined,  and  contractors  currently  consult  with 
schools  on  the  appropriate  IMI  levels  for  varying  pieces  of  content.  Moreover,  ATSC  has 
developed  a  series  of  checklists  for  schools  that  may  have  helped  them  improve  processes 
related  to  IMI  quality. 

19  The  specific  question  was  “How  satisfied  or  dissatisfied  are  you  with  your  school’s  cur¬ 
rent  IMI  development  program  with  regard  to  the  quality  of  DL  products?”  For  a  complete 
description  of  the  school  interviews,  see  Appendix  A. 
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approach  to  instruction,  to  gain  the  buy-in  of  DL  stakeholders.  Given 
the  recent  sharp  decline  in  the  overall  budget  for  TADLP,  a  clear  record 
of  success  may  be  a  prerequisite  for  securing  the  resources  needed  to 
achieve  even  a  modest  number  of  the  program’s  goals. 

Recommendations  for  Improvement 

We  recommend  that  a  program-level  evaluation  of  IMI  content  be 
established  within  TADLP.  Such  a  program  could  be  instituted  in  sev¬ 
eral  stages,  as  described  below. 

Establish  a  program-level  capability  for  content  evaluation.  We 

recommend  that  TRADOC  establish  the  capability  to  undertake  a 
more  extensive  content  evaluation  of  at  least  a  sample  of  TADLP  les¬ 
sons  as  they  become  available.  As  explained  in  Chapter  Three,  a  content 
evaluation  involves  the  assessment  of  training  materials  using  external 
evaluators.  Several  criteria  are  recommended,  including  those  listed  in 
Table  4.6. 

TRADOC  could  establish  program-level  content  evaluation  with 
relatively  modest  resources  by  using  a  sampling  strategy  (i.e.,  evaluate 
a  sample  of  lessons  rather  than  entire  courses).  Once  criteria  are  devel¬ 
oped  and  raters  are  trained,  lessons  can  be  evaluated  with  relative  effi¬ 
ciency.  A  structured  research  effort  like  that  undertaken  in  our  project 
could  be  used  to  help  determine  the  success  of  new  initiatives  aimed 
at  improving  quality  and  to  guide  the  Army  toward  the  most  effective 


Table  4.6 

Criteria  for  Program-Level  Training  Content  Evaluation 


Technical 

Production  Quality 

Pedagogical 

•  Launching 

•  Legibility  of 

•  Learning  objectives 

•  Navigation,  tracking. 

graphics  and  text 

•  Sequencing,  pacing,  and 

and  orientation 

•  Audiovisuals 

learner  control 

•  Supplementary 

•  Feedback 

instructional  materials 

•  Instruction  of  concepts 

•  Technical  support 

•  Instruction  of  processes 

•  Instruction  of  procedures 

•  Checks  on  learning 

•  Practical  exercises 
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use  of  IMI  for  Army  training  by  evaluating  existing  courses  to  identify 
ways  to  improve  them.  Moreover,  this  type  of  effort  could  support  a 
move  toward  a  “best  value”  approach  to  contractor  selection,  as  recom¬ 
mended  in  the  first  section  of  this  chapter. 

The  results  of  our  pilot  indicate  that  a  content  evaluation  of 
courseware  is  feasible,  can  provide  the  Army  with  valuable  informa¬ 
tion  about  courseware  quality,  and  point  to  directions  for  needed  qual¬ 
ity  improvements.  The  method  we  used  has  a  variety  of  strengths.  It 
provides  a  systematic  method  of  evaluation  using  multiple  raters  and  a 
comprehensive  set  of  criteria  based  on  standards  proposed  by  experts  in 
training  development  and  assessment.  Moreover,  these  criteria  enable 
integration  of  results  across  courses,  schools,  or  other  units  of  analy¬ 
sis.  Use  of  this  approach  to  evaluation  could  be  strengthened  further; 
therefore,  we  recommend  that  the  Army: 

•  Evaluate  fully  functional  courseware  rather  than  courseware  pro¬ 
vided  on  CD,  and  reassess  and  refine  the  criteria  to  reflect  fully 
operational  courseware. 

•  Where  possible,  establish  objective  standards  for  the  criteria  such 
as  the  degree  to  which  lessons  provide  sufficient  examples,  checks 
on  learning,  and  practical  exercises. 

•  Establish  weights  for  the  criteria  according  to  their  relevance  to 
course  objectives  and  the  resource  implications  of  providing  fixes 
for  deficiencies. 

•  Evaluate  online  versions  of  courses  in  order  to  include  an  evalua¬ 
tion  of  additional  features,  such  as  course  tests,  bookmarks,  and 
“live”  technical  support. 

Establish  a  program-level  capability  for  additional  types  of 
IMI  quality  assessments.  A  content  evaluation  covers  some  aspects  of 
training  quality,  but  a  comprehensive  assessment  requires  a  multidi¬ 
mensional  approach.  In  addition,  a  number  of  approaches  to  evaluat¬ 
ing  quality,  discussed  in  Chapter  Three  of  this  report,  can  be  facilitated 
by  the  use  of  information  technology  (IT).  Table  4.7  lists  some  of  the 
kinds  of  evaluations  that  could  be  performed  by  the  Army. 
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Table  4.7 

Types  of  IMI  Evaluations  That  Might  Be  Performed  by  the  Army 


Type  of  Evaluation 


What  Could  Be  Done? 


Learner  reactions 


Learning:  pre/post 
comparisons 

Learning:  knowledge 
retention 

Performance  (predictive 
validity) 


Test  evaluation 


Indirect  metrics 


Develop  core  set  of  questions  for  trainees  across  IMI 
courses 

Develop  platform  to  enable  schools  to  create/ 
customize  surveys 

Create  automatic  scoring  and  reporting  capabilities 

Develop  platform  and  automated  scoring/reporting 
procedures  to  support  analysis  across  courses 

Administer  follow-up  tests  to  trainees 

Assess  strengths  and  weaknesses  of  AUTOGEN; 
explore  facilitators  and  barriers  to  using  it  to  collect 
performance  data 

Make  item  analysis  of  end-of-course  tests  an  integral 
part  of  IMI  via  the  LMS 

Expand  and  standardize  RAND's  initial  analyses 


A  broad-based  assessment  might  include  the  measures  of  student 
outcomes  listed  above,  many  of  which  could  be  tested  on  a  pilot  basis 
using  a  sample  of  IMI  courses: 

•  Learner  reactions.  The  Army  might  develop  a  core  set  of  ques¬ 
tions  to  administer  to  learners  across  IMI  courses,  develop  an  IT 
platform  to  enable  schools  to  create/customize  surveys,  and  create 
automatic  scoring  and  reporting  capabilities. 

•  Learning  (pre/post  comparisons).  The  Army  could  develop  an  IT 
platform  to  administer  course  pretests  and  posttests.  Automatic 
scoring/reporting  procedures  could  support  systematic  analysis  of 
test  scores  across  courses  or  schools. 

•  Learning  (knowledge  retention).  The  Army  could  administer 
and  score  follow-up  tests  relatively  efficiently  and  inexpensively 
using  IT.  Such  tests  could  be  administered  after  learners  return  to 
their  units  or  when  they  register  for  or  attend  the  resident  portion 
of  a  course  after  completing  the  DL  phase. 

A  broad-based  assessment  might  also  include  the  other  approaches 
listed  in  the  table: 


Improvements  to  the  IMI  Program  91 


•  Performance  (predictive  validity).  IT  could  be  used  to  collect 
and  analyze  data  to  assess  the  predictive  validity  of  training,  i.e., 
the  association  of  individuals’  performance  in  DL  with  perfor¬ 
mance  in  subsequent  residential  training  or  with  ratings  of  subse¬ 
quent  job  performance  in  the  unit. 

•  Test  evaluation.  IT  can  be  used  to  administer  course  tests  and 
conduct  statistical  analysis  of  objective  test  items  (i.e.,  multiple 
choice  or  true/false)  to  provide  information  such  as  whether  items 
are  at  the  appropriate  level  of  difficulty  and  whether  the  tests  dis¬ 
criminate  among  good  and  poor  performers  in  the  course. 

•  Indirect  metrics.  Automated  systems  can  capture  data  such  as 
enrollment  and  dropout  rates,  DL  usage,  and  information  per¬ 
taining  to  course  development,  such  as  cycle  time.  These  types  of 
indirect  metrics  can  be  indicators  of  course  quality  (e.g.,  DL  usage 
might  be  low  because  courses  are  of  low  quality)  and  can  also  be 
used  to  help  keep  the  program  on  track  with  regard  to  TADLP 
objectives. 

Assessments  should  eventually  extend  beyond  quality.  In  a 

broader  context,  evaluation  of  IMI  quality  should  be  part  of  a  more 
comprehensive  assessment  component  for  TADLP.  In  addition  to  eval¬ 
uating  quality,  that  program  would  examine  program  impact  bene¬ 
fits,  efficiency,  and  cycle  time  issues.  Additionally,  assessment  would 
involve  not  only  where  improvement  is  needed,  but  also  an  effort  to 
determine  underlying  causes  of  unsatisfactory  outcomes,  and  possible 
directions  for  improvement.  Taken  together,  these  efforts  will  give  the 
Army  a  basis  for  managing  continuous  improvement  in  the  develop¬ 
ment  and  use  of  IMI,  and  provide  important  tools  to  help  meet  the 
goals  of  TADLP. 


Chapter  Conclusions 

The  five  recommendations  described  in  this  chapter  are  directed  toward 
improvement  in  the  IMI  development  program  run  by  TRADOC  in 
support  of  DL.  Each  of  these  measures  is  likely  to  have  either  direct  or 
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indirect  effects  on  the  measures  of  effectiveness  described  in  Chapter 
Three.  Table  4.8  lists  the  five  recommendations  in  the  first  column, 
then  indicates  for  each  measure  of  effectiveness  whether  the  improve¬ 
ment  would  likely  have  a  “direct”  benefit  or  a  “derivative”  or  indirect 
benefit. 


Table  4.8 

Likely  Effect  of  Recommendations  on  Measures  of  Effectiveness 


Recommendation 

Program 

Impact 

Efficiency 

Quality 

Cycle 

Time 

Responsive¬ 

ness 

New  acquisition 
strategy 

Derivative 

Direct 

Direct 

Direct 

Direct 

More  resources 
per  hour 

Derivative 

Derivative 

Direct 

Direct 

Direct 

Systematic 
process 
improvement 
of  cycle  time 

Direct 

Direct 

Derivative 

Direct 

Direct 

Increase  local 
participation  in 

IMI  production 

Derivative 

Direct 

Direct 

Direct 

Direct 

Begin  program- 
level  quality 
evaluation 

Derivative 

Derivative 

Direct 

Derivative 

Direct 
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Transforming  TADLP  to  Better  Support  Training 
and  Leader  Development  Requirements 


In  the  previous  chapters  of  this  report,  we  have  presented  the  results  of 
our  examination  of  TADLP  and,  in  particular,  the  Army’s  approach  to 
IMI.  This  examination  led  to  a  discussion,  in  Chapter  Four,  of  recom¬ 
mendations  that  can  be  implemented  to  improve  the  Army’s  existing 
approach  to  IMI.  However,  moving  to  the  third  purpose  of  the  project 
and  stage  of  the  analysis  (see  Figure  1.2),  the  Army  also  needs  to  con¬ 
sider  broader,  strategic  changes  in  its  approach  to  DL.  Such  changes 
are  necessitated  by  the  increasing  need  for  DL,  in  terms  of  the  number 
of  DL  courses  provided,  the  kinds  of  skills  (e.g.,  complex  thinking  and 
problem  solving)  trained  through  DL,  and  the  scheduling  of  institu¬ 
tional  training  to  support  unit  readiness  cycles. 

This  chapter  addresses  our  first  recommendation  for  broaden¬ 
ing  TADLP  beyond  a  primary  focus  on  IMI  delivery.  Specifically,  it 
describes  ongoing  initiatives  and  other  options  to  expand  TADLP  to 
meet  the  Army’s  future  training  needs.  Options  focus  particularly  on 
the  development  of  a  blended  learning  approach.  Before  describing 
these  initiatives,  we  briefly  make  the  case  for  transformational  change 
to  TADLP. 
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New  Demands  on  Training  and  Leader  Development 
Point  to  Need  for  Transformational  Change 

As  discussed  in  Chapter  One,  developing  and  executing  training  pro¬ 
grams  has  become  an  ever  more  challenging  task,  and  the  growing 
demands  on  training  and  leader  development  strategies  in  turn  increase 
the  demand  on  institutional  training.  There  have  also  been  pressures  to 
reduce  resident  course  length,  and  this  generates  a  need  for  expanding 
TADLP. 

Perhaps  the  largest  challenge  is  posed  by  the  need  to  prepare 
leaders  for  a  greater  range  of  possible  operational  missions,  or  full- 
spectrum  operations,  while  simultaneously  supporting  a  demand¬ 
ing  set  of  ongoing  operations.  This  challenge  is  made  more  difficult 
because  the  training  system  is  already  struggling  to  cope  with  several 
other  new  demands:  modularity  and  modernization.1  These  changes 
have  meant  that  the  Army  needs  to  prepare  soldiers  and  leaders  to 
take  on  a  range  of  increased  and  continually  changing  requirements. 
The  need  for  enhanced  leader  development  is  especially  important. 

At  the  same  time,  ARFORGEN,  the  Army’s  cyclical  process  for 
preparing  and  providing  forces  for  deployment,  requires  Army  units 
to  move  through  a  structured  set  of  collective  training  exercises  over  a 
specified  training  cycle,  ideally  set  at  three  years  for  the  AC,  but  shorter 
in  current  practice.  ARFORGEN  puts  great  demands  on  the  schedul¬ 
ing  of  institutional  courses  and  limits  the  length  of  time  soldiers  can 
spend  at  a  schoolhouse  away  from  home  station. 

As  an  example  in  terms  of  NCOES,  the  impact  of  ongoing  opera¬ 
tions  has  been  especially  significant.  Large  numbers  of  NCOs  simply 
have  not  been  able  to  go  to  the  courses  developed  to  prepare  them  for 
their  duty  positions.  As  of  August  2008,  about  50,000  NCOs,  a  quar¬ 
ter  of  the  Army’s  NCO  population,  were  in  such  a  backlog  status.2  For 
example,  about  19,600  AC  staff  sergeants  (out  of  a  population  of  just 
over  64,400)  had  not  completed  BNCOC. 


1  These  concepts  are  discussed  in  more  detail  in  Chapter  One. 

2  Data  obtained  from  the  DA  G-l  staff  in  September  2008. 
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Additionally  many  NCOs  are  assigned  to  positions  authorized 
to  be  filled  by  a  higher  grade  (e.g.,  sergeants  being  assigned  to  a  staff 
sergeant  position)  and  would  normally  not  have  attended  the  course 
required  for  the  higher  grade.3  Thus,  it  seems  likely  that  a  third  to  half 
of  today’s  NCOs  have  not  attended  the  courses  designed  to  give  them 
the  skills  and  training  to  serve  in  their  current  duty  positions.  Given 
the  expectations  of  continued  but  reduced  operational  demands,  even 
if  deployments  can  be  reduced  to  one  year  out  of  three,  getting  NCOs 
to  readiness-required  courses  is  likely  to  remain  a  challenge. 

The  needs  and  requirements  just  described  have  led  the  Army  to 
place  increased  emphasis  on  DL.  For  example,  the  draft  Army  Cam¬ 
paign  Plan  outlines  a  major  role  for  TADLP  in  supporting  a  lifelong- 
learning  construct.  In  particular,  the  Army  has  been  aiming  to  leverage 
the  potential  of  DL  to  cover  newly  required  skills  and  tasks,  envision¬ 
ing  a  major  role  for  DL  in  transforming  its  training  and  leader  develop¬ 
ment  programs. 

But  the  budgets  to  support  DL  initiatives  have  been  limited  and, 
as  we  have  shown  earlier  in  this  report,  are  decreasing.  Further,  the 
amount  of  training  courseware  and  the  training  benefit  from  the  DL 
courseware  that  has  been  produced  and  kept  in  use  have  been  modest 
and  focused  on  the  RC.  So  while  the  DL  program  has  been  helpful, 
it  has  provided  only  limited  increases  in  the  amount  of  training  and 
leader  development  accomplished.4  Clearly,  if  the  heightened  roles 
envisioned  for  TADLP  are  to  be  achieved,  major  program  improve¬ 
ment  is  needed. 

Better  leveraging  DL  could  conceivably  provide  many  benefits. 
First,  this  approach  could  facilitate  scheduling  training  to  fit  unit  win¬ 
dows,  thereby  benefiting  unit  readiness,  because  more  leaders  would 
enter  collective  training  with  better  preparation.  Second,  the  approach 


3  This  is  based  on  ongoing  RAND  research  that  examined  the  grade  of  NCOs  assigned 
for  unit  positions  during  preparation  training  events  for  deployment.  Results  of  the  analysis 
showed  that  mismatches  were  frequent.  For  example,  almost  a  quarter  of  the  squad  and  sec¬ 
tion  leader  positions  were  filled  by  NCOs  of  a  lower  grade  than  the  authorized  level  of  staff 
sergeant. 

4  Based  on  an  ongoing  RAND  Arroyo  Center  examination  of  DL  courses  and  graduates 
across  the  last  several  years. 
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also  has  the  potential  to  reduce  costs  by  eliminating  the  need  to  send 
soldiers  to  resident  courses. 


Blended  Learning  Options  Have  the  Potential  to 
Significantly  Expand  DL's  Role 

The  Army's  Approach  to  DL  Has  Limitations 

Up  through  the  time  of  our  assessment,  the  Army  has  used  what  might 
be  called  a  “stand-alone”  IMI  model  as  its  dominant  approach  to  DL. 
Under  this  model,  contractors  are  used  to  develop  40+  hour  chunks  of 
custom  IMI  content  that  is  designed  to  require  little  or  no  instructor  or 
other  support  during  delivery. 

This  approach,  by  itself,  has  turned  out  to  be  severely  limited  in 
its  ability  to  fulfill  the  Army’s  expanded  need  for  timely  and  export¬ 
able  training.  First,  as  described  in  Chapter  Three,  the  courseware 
development  takes  too  long  to  meet  many  of  today’s  needs.  Further, 
given  the  Army’s  decreasing  budget  for  courseware  and  the  relatively 
small  throughput  for  many  courses,  the  Army’s  estimated  average  rate 
of  $14,000  per  training  hour  for  DL  conversion  implies  that  relatively 
few  courses  will  be  converted  and  that  payback  periods  for  the  initial 
investment  will  be  long.  Finally,  although  the  new  proposed  acquisi¬ 
tion  strategy  for  IMI  has  potential  to  increase  the  responsiveness  of 
courseware  to  needed  changes,  the  time  required  to  implement  changes 
to  a  DL  course  would  still  be  far  longer  than  the  time  needed  to  alter 
a  course  that  is  exported  by  other  means  (e.g.,  a  mobile  training  team 
(MTT)  or  VTT). 

The  stand-alone  IMI  model  is  also  limited  in  other  ways  in  its 
ability  to  meet  all  the  Army’s  DL  needs.  For  example,  stand-alone,  self- 
paced  IMI  can  be  used  cost-effectively  only  to  provide  lower  levels  of 
learning  (i.e.,  knowledge,  comprehension),5  as  opposed  to  the  higher 
levels  of  learning  (i.e.,  analysis,  synthesis,  and  evaluation)  needed  for 
most  leader  tasks.  Put  another  way,  stand-alone  IMI  can  often  teach 


5  Bloom  (1964);  see  also  a  summary  at 
http://www.nwlink.com/^donclark/hrd/bloom.html. 
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enabling  learning  objectives  (ELOs)  from  a  course,  but  not  often  the 
terminal  learning  objectives  (TLOs).  For  example,  while  IMI  can  be 
used  to  teach  knowledge  of  the  tactics  involved  in  a  platoon  attack,  it 
cannot  exercise  or  test  a  student’s  ability  to  develop  an  actual  attack 
plan  without  the  development  of  complex,  expensive  courseware.  As 
a  result,  the  Army’s  dominant  model  of  DL  delivery  is  limited  from 
the  start,  since  few  terminal  learning  objectives  or  tasks  can  truly  be 
trained  to  standard  using  just  stand-alone  IMI. 

The  Army’s  implementation  of  IMI  training  is  also  limited.  For 
example,  in  the  Army’s  phased  approach  to  DF  (where  DF  is  preceded 
by  a  residential  phase),  there  is  often  an  extended  period  between  learn¬ 
ing  the  EFOs  through  IMI  and  completing  the  TFOs  in  the  residential 
part  of  a  phased  course,  thus  making  the  potential  for  learning  decay 
an  important  factor.  Most  of  the  advantages  of  having  an  IMI  prepara¬ 
tory  phase  disappear  entirely  if  instructors  have  to  review  in  residence 
the  material  taught  in  the  IMI.  Also,  the  relatively  long  IMI  courses 
combined  with  the  lack  of  regular  instructor  contact  likely  mean  that 
the  completion  rate  for  DF  courses  will  lag  considerably  behind  what 
is  achieved  in  other  courses. 

As  a  result,  the  Army’s  IMI  stand-alone  model,  even  with  imple¬ 
mentation  of  the  improvements  in  IMI  development  described  in 
Chapter  Four,  is  unlikely  to  satisfy  the  Army’s  large  need  for  expanded 
exportable  training  in  NCOES  and  other  key  readiness-enabling 
courses  required  during  the  Reset  phase  of  the  ARFORGEN  cycle. 

Additional  Options  Expand  the  Range  of  DL  Benefits 

To  meet  increasing  demands  for  DF,  the  Army  will  need  to  expand 
its  traditional  approach  to  developing  and  delivering  DF.  Below  we 
describe  the  Army’s  approach  compared  to  six  other  options  for  DF. 
We  highlight  and  compare  some  of  the  salient  advantages  and  disad¬ 
vantages  of  each  approach. 

IMI  Levels  1-3,  Stand  Alone.  The  Army’s  approach  of  using 
stand-alone  IMI  with  interactivity  Fevels  1-3  has  the  benefit  of  pro¬ 
viding  scheduling  flexibility  to  the  student,  who  can  access  the  course 
24  hours  a  day,  7  days  per  week.  Further,  because  the  instruction  is 


98  Making  Improvements  to  The  Army  Distributed  Learning  Program 


embedded  in  the  technology,  limited  interaction  or  collaboration  with 
instructors  is  needed. 

However,  for  the  same  reasons,  the  method  also  requires  a  signifi¬ 
cant  number  of  training-development  hours  to  produce.  The  training- 
development  time  required  to  produce  one  course  hour  of  IMI  content 
(mix  of  Levels  1-3)  is  estimated  to  be  about  220  hours.6  As  described 
above,  there  are  major  disadvantages  to  this  approach,  most  notably 
that  instruction  can  cost-effectively  be  used  to  provide  only  knowledge 
and  comprehension  (and  sometimes  application)  levels  of  learning,  not 
the  higher  levels  of  analysis,  synthesis,  and  evaluation.7  As  a  result, 
other  modalities  of  learning  would  be  required  for  students  to  reach 
terminal  learning  objectives  on  most  key  tasks  in  leader  courses. 

IMI  Levels  1-3,  Instructor  Supported.  This  approach  similarly 
depends  on  IMI  technology  to  deliver  the  content,  but  also  makes 
human  instructors  available  to  monitor  and  support  learner  progress, 
respond  to  student  questions  about  content,  and  provide  feedback  on 
tests.  This  capability  must  be  supported  through  communication  and 
collaboration  technologies  (described  below)  and  is  not  generally  fea¬ 
sible  unless  the  student  is  taking  the  learning  online.  Depending  on 
the  level  of  instructor  support,  this  approach  often  provides  less  flex¬ 
ibility  to  the  learner  than  other  DL  approaches  do  because  student-to- 
instructor  communication  is  usually  asynchronous  and  requires  wait¬ 
ing  for  a  response  (e.g.,  a  course  may  be  designed  so  that  instructors 
respond  to  learners  within  24  hours  of  a  question  or  request). 

However,  this  method  can  support  somewhat  higher  levels  of 
learning  compared  to  stand-alone  IMI  Levels  1-3  because  of  the  added 
instructional  support.  In  particular,  instructors  end  up  providing  moti¬ 
vation  as  well  as  diagnostic  and  prescriptive  instruction  to  help  indi¬ 
viduals  understand  concepts  and  acquire  skills.  Finally,  because  of  the 
instructor  role  in  providing  the  training,  the  IMI  itself  can  be  more 
basic  (e.g.,  involve  less  interactivity)  and  less  comprehensive  in  some 
areas  compared  to  what  is  required  for  stand-alone  IMI,  allowing  for 


6  Chapman  et  al.,  (2006b).  This  estimate  also  corresponds  fairly  closely  to  TRADOC 

estimates. 
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Shanley  et  al.  (2007),  p.  190. 
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a  somewhat  reduced  training  development  effort  up  front  to  create 
content.8 

IMI  Level  4,  Asynchronous.  IMI  Level  4  usually  consists  of  simu¬ 
lations  or  “serious  games”  to  provide  learners  with  an  immersive  expe¬ 
rience  at  high  levels  of  interactivity  in  the  learning  domain  to  facilitate 
TLO-level  training.  This  approach  typically  provides  the  same  flexibil¬ 
ity  of  access  as  found  in  any  level  of  IMI,  while  allowing  the  teaching 
of  richer  knowledges  and  experiences  than  found  at  lower  interactivity 
levels.  Highly  interactive  experiences  can  be  used  as  a  motivating  and 
orienting  tool  and  can  also  provide  for  introduction  of  more  complex 
content. 

However,  achieving  this  level  of  interactivity  often  requires  sig¬ 
nificantly  higher  levels  of  training  development  investment  per  hour 
of  content  compared  to  lower  levels  of  interactivity  or  instructor-led 
training.9  Moreover,  while  this  approach  is  potentially  effective  for 
increasing  some  skill  levels,  it  would  likely  have  too  high  a  develop¬ 
ment  cost  (in  terms  of  both  SME  and  technical  expertise)  to  stand 
alone  in  providing  positive  “training  transfer”10  on  most  tasks.  Further, 
this  approach  is  usually  appropriate  only  for  a  very  limited  part  of  most 
programs  of  instruction. 

IMI  Level  4,  Synchronous.  This  approach  usually  means  synchro¬ 
nous  simulations  and  serious  games  involving  multisided  interactions 
with  two  or  more  players  on  opposing  sides,  or  involving  multiple  role 
players,  as  well  as  an  instructor  to  oversee  and  facilitate  an  end  of  exer¬ 
cise  After-Action  Review  (AAR).  This  technology  is  offered  mainly  as 
a  component  of  resident  training,  but  the  capability  for  online  virtual 


8  In  fact,  the  content  can  sometimes  be  reduced  to  PowerPoint  slides  with  voice  narration. 
Studies  have  shown  (see  Chapman  reference  above)  that  this  approach  to  IMI  requires  no 
more  training  development  effort  than  instructor-led  training. 

9  One  estimate  is  that  it  takes  750  hours  of  simulation  development  time  to  create  an  hour 
of  training,  compared  with  34  hours  of  development  time  to  create  an  hour  of  instructor- 
led  training  and  220  to  create  an  hour  of  IMI,  Levels  1-3.  See  B.  Chapman  et  ah.  Online 
Simulations:  A  Knowledgebase  of  100+  Simulation  Development  Tools  and  Services,  Sunnyvale, 
Calif.:  Brandon  Hall  Research,  2006a. 

10  To  achieve  positive  training  transfer,  the  training  has  to  provide  exercises  that  teach  the 
appropriate  actions  in  a  real-world  activity. 
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application  has  been  available  for  many  years  and  is  undergoing  great 
growth  in  both  rhe  commercial  and  military  sectors.  There  is  a  great 
potential  application  for  this  type  of  technology  to  provide  for  instruc¬ 
tion  of  more  complex  content  at  higher  levels  of  instruction,  such  as 
aspects  of  analysis,  synthesis,  and  evaluation.  Further,  it  offers  a  flexible 
means  to  change  scenarios  and  does  not  place  significant  demands  on 
artificial  intelligence  capabilities  to  provide  for  realistic  outcomes.  A 
disadvantage  of  this  approach  is  that  it  requires  significant  investment 
in  training  development  resources  (although  less  than  for  asynchro¬ 
nous  Level  4  IMI,  given  that  less  artificial  intelligence  for  player  actions 
is  required).  Another  disadvantage  is  that  it  requires  significant  effort 
for  scheduling  and  coordination. 

Asynchronous  Collaborative  DL.  This  approach  uses  technology 
to  enable  communication  between  student  and  instructor  and  among 
students  via  such  avenues  as  email,  forums,  discussion  boards,  tele¬ 
phones,  or  wikis.  It  involves  considerably  more  interaction  than  the 
instructor-supported  IMI  described  above.  No  matter  which  method 
is  used,  the  instruction  comes  from  the  instructor,  while  the  role  of 
technology  is  one  of  facilitation.  Since  the  primary  support  for  the 
instruction  is  provided  by  the  human  instructor,  this  approach  to 
DL  can  be  used  to  teach  all  levels  of  knowledge.  For  example,  under 
this  approach  the  student  could  be  required  to  write  an  attack  order, 
submit  it  to  the  instructor  for  review,  and  than  receive  detailed  feed¬ 
back.  This  approach  can  also  provide  for  student-to-student  as  well  as 
student-to-instructor  interaction,  and  according  to  some  developers  we 
interviewed,  the  levels  of  interaction  can  be  higher  than  in  traditional 
classroom  approaches.  As  with  asynchronous  instruction  in  the  IMI 
context  (explained  above),  the  flexibility  of  asynchronous  collabora¬ 
tive  DL  is  moderate — the  learners  typically  have  delayed  access  to  the 
instructor  and  fellow  students.  Training  development  costs  are  often 
relatively  low  because  basic  types  of  IMI  can  often  be  used  for  support, 
but  instructor  support  is  often  substantial,  sometimes  at  the  same  level 
as  provided  in  residential  instruction. 

Use  of  this  method  depends  on  collaborative  technologies  to 
enable  the  interactions  and  exchange  of  documents,  and  may  require 
some  training  so  that  participants  have  sufficient  command  of  those 
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technologies.  There  are  also  often  requirements  for  common  features 
of  learning  management  systems,11  such  as  online  posting  of  assign¬ 
ments,  online  grade  books,  time-stamped  submission  of  assignments, 
and  online  quizzes  and  tests. 

Synchronous  Collaborative  DL.  This  approach  involves  real-time 
interaction  between  students  and  instructors.  Like  instructor-centric 
asynchronous  collaborative  DL,  this  method  requires  the  instructor 
to  carry  the  weight  of  the  instructional  delivery,  and  it  can  be  used  to 
teach  all  levels  of  knowledge.  However,  having  students  and  instructor 
online  at  the  same  time  can  greatly  increase  the  difficulties  involved 
in  scheduling  and  coordination.  Training  development  costs  are  com¬ 
parable  to  those  needed  for  residential  training,  but  travel  costs  are 
obviously  reduced  greatly.  Delivery  costs  will  depend  on  the  level  of 
dispersion  among  learners  and  the  need  for  down-site  instructors  or 
coordinators.  It  would  also  require  either  compatible  facilities,  or  provi¬ 
sion  of  appropriate  technology  to  students. 

Web-based  synchronous  and  collaborative  DL  requires  its  own 
set  of  technologies  to  enable  delivery.  Commercial  tools  are  available  to 
provide  many  of  the  features  common  to  this  approach:12 

•  Many  simultaneous  participants  in  a  web-based  “classroom.” 

•  Instructor’s  ability  to: 

-  Broadcast  voice  or  video  to  participants. 

-  Work  through  briefing  slides,  demonstrate  software  applica¬ 
tions  running  locally  on  a  computer  or  on  the  web,  show  videos. 

-  Patch  in  live  video  streams  from  other  locations. 

-  Take  questions  (via  voice,  chat,  or  other  method  of  submission). 

-  Provide  voice  communication  by  individual  participants  to  all 
participants  to  share  an  insight  or  question. 

•  Participants’  ability  to: 

-  Take  control  of  applications  or  collaboration  skills. 

-  Pose  questions  to  the  instructor. 

-  Chat  with  each  other. 


11  Rossett  (2002). 

12  Woodhill  (2007). 
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Some  collaborative  learning  tools  also  allow  the  instructor  to 
record  the  entire  live  session  into  a  digital  movie  format  that  can  later 
be  used  to  provide  an  asynchronous  learning  experience.  All  voice 
interactions,  images  that  appear  on  the  screen,  participant  questions, 
and  other  interactions  are  part  of  that  recorded  session. 

Mobile  Training  Teams.  MTTs  provide  instruction  to  learners  at 
the  learners’  location  rather  than  at  the  schoolhouse.  Variations  on  this 
approach  include  the  use  of  local  schools  at  learners’  home  stations  or 
instruction  conducted  by  unit  leaders  with  the  support  of  institutional 
resources  (e.g.,  training  support  packages  or  TSPs). 

Though  technically  not  classified  as  a  DL  approach  to  instruction, 
the  use  of  MTTs  is  an  important  method  of  exporting  training  that  the 
Army  employs.  Its  primary  advantage  over  other  instructor-supported 
methods  is  that  it  provides  the  opportunity  for  hands-on  instruction 
and  practical  exercises  in  the  held  or  on  equipment  that  is  not  possible 
with  other  methods.  Further,  this  approach,  without  using  technology, 
is  often  faster  and  more  effective  at  instruction  at  higher  learning  levels 
than  other  technology-intensive  learning  methods.  The  primary  limita¬ 
tion  of  this  method  is  that  it  is  practical  only  when  there  is  a  sufficient 
number  of  learners  concentrated  in  one  location. 

Many  Organizations  Mix  DL  Approaches  Within  a  Course  to  Form 
Blended  Learning  Options 

Many  organizations  using  DL  are  moving  toward  employing  a  mix 
of  approaches  within  a  single  course.  This  new  type  of  course  is  con¬ 
structed  by  considering  all  potential  modalities  for  delivering  training 
on  a  task-by-task  basis,  leading  to  a  distribution  of  different  modali¬ 
ties  across  the  entire  course.  Examples  of  possible  combinations  of  DL 
modalities  are  shown  in  Figure  5.1.  As  implied  by  the  previous  discus¬ 
sion  and  the  top  part  of  the  figure,  the  Army  has  traditionally  made  use 
of  only  a  small  number  of  the  possibilities  within  a  course,  for  example, 
using  stand-alone  IMI  to  take  the  place  of  an  average  of  60  hours  of 
instruction  in  a  much  longer  course.  VTT  and  MTTs  have  also  been 
used.  In  contrast,  many  organizations  needing  to  export  training  are 
making  choices  that  involve  a  more  complex  combination  of  the  avail- 
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Figure  5.1 

Current  and  Future  Possible  Combinations  of  DL  and  Other  Training 
Approaches 
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able  options,  leading  to  the  creation  of  blended  learning  courses.  These 
are  portrayed  by  the  examples  in  the  bottom  part  of  the  figure.13 

Making  decisions  about  which  modalities  best  train  individ¬ 
ual  objectives  is  a  difficult  undertaking.  The  general  considerations 
described  in  the  previous  subsection  regarding  the  varying  capabilities, 
constraints,  and  resource  requirements  of  DL  approaches  provide  a 
host  of  factors  that  need  to  be  taken  into  account  in  specific  instances. 
Important  factors  include  the  following: 

•  Expected  usage  rates  of  the  training  content. 

•  Deadline  for  delivering  the  DL. 

•  Speed  with  which  TLO  can  be  achieved  (if  the  DL  only  addresses 
ELOs). 

•  Stability  of  the  training  content. 

•  Degree  of  dispersion  of  learners  requiring  the  training. 


13  For  examples,  see  Chapman  (2008). 
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•  Need  for  student-to-instructor  and  student-to-student  interaction. 

•  Supply  of  instructors  with  the  required  experience. 

•  Need  for  equipment  or  training  aids. 

Moreover,  decisionmaking  is  made  more  difficult  by  the  rapid  increase 
in  the  capabilities  of  collaborative  tools  for  enabling  DL  instruction. 
The  Army  has  no  effective  guidance  and  few  tools  or  processes  to  help 
determine  the  best  mix  of  approaches  to  maximize  the  effectiveness  of 
its  modest  budget  for  courseware.  Data  from  the  commercial  and  aca¬ 
demic  sectors  indicate  that  determining  the  best  mix  is  becoming  more 
complicated  and  varies  by  subject  matter  and  industry.14 

Award-Winning  Commercial  Companies  Have  Dramatically 
Increased  the  Exportability  of  Leadership  Training  Using  Blended 
Learning  Models 

In  Figure  5.2  we  provide  an  example  from  the  commercial  sector  of 
how  training  can  be  successfully  exported.  The  example  illustrates 
IBM’s  blended  learning  program,15  which  earned  the  company  the 
“Technology  in  Action”  award  from  Training  magazine  and  caused  the 
company  to  be  recognized  as  a  “training  Top  100  company”  for  over 
five  years.16 

IBM  is  the  world’s  largest  information  and  technology  company, 
with  over  300,000  employees  in  75  countries.  The  company’s  train¬ 
ing  focuses  on  leadership  and  management  as  well  as  technical  skills, 
and  involves  the  creation  of  global  virtual  teams.  The  training  often 
deals  with  complex  issues,  e.g.,  issues  of  re-skilling  the  workforce  to 
address  key  market  shifts  and  organizational  transformations.  On  aver¬ 
age,  employees  average  55  hours  of  training  a  year.  As  in  the  Army, 
IBM  personnel  have  decreasing  amounts  of  time  available  for  training 
and  formal  training  budgets  are  limited.  IBM  needed  a  capability  that 
would  allow  employees  to  learn  anywhere,  anytime,  and  on  demand. 


14  Wexler  (2008). 

15  Chapman  (2008)  and  “IBM’s  Learning  Transformation  Story”  (2004). 

16  Training  determines  the  award  winner  in  a  multitiered  nomination,  application,  and 
interview  process.  IBM  has  been  among  the  top  five  candidates  for  five  years  running. 
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Figure  5.2 

IBM's  Blend  of  Learning  Approaches  Minimizes 
Resident  Contact  Hours 
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IBM  transformed  its  formal  learning  program  by  moving  from 
a  model  based  primarily  on  classroom  training  to  a  blended  learning 
model  that  optimizes  a  combination  of  DL  approaches  to  export  as 
much  of  the  training  as  possible.  IBM  settled  on  using  the  following 
mix  of  learning  modalities  to  achieve  their  optimal  combination. 

•  Web-based  IMI,  supported  by  an  instructor.  Used  to  instill  facts, 
figures,  and  other  knowledge  about  the  subject,  as  well  as  compre¬ 
hension  of  concepts  and  procedures.  Training  content  can  include 
documents  as  well  as  basic  distributed  learning  instruction.  This 
approach  accounted  for  50  percent  of  training  hours. 

•  Asynchronous  simulation  and  games.  Used  to  provide  immer¬ 
sive  experience  and  allow  role-playing  for  problem  solving.  This 
approach  was  not  designed  to  reach  terminal  learning  objectives, 
but  instead  was  to  motivate  and  familiarize  learners  with  the  con¬ 
texts  within  which  they  will  apply  the  information  they  are  learn¬ 
ing.  This  approach  accounted  for  15  percent  of  training  hours. 
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•  Collaborative,  asynchronous  and  synchronous.  Includes  use  of 
virtual  classrooms,  e-labs,  online  conferences,  and  other  technolo¬ 
gies  to  team  students  up  for  group  projects  and  provide  for  stu- 
dent-to-student  as  well  as  student-to-instructor  interactions.  This 
approach  accounted  for  15  percent  of  training  hours. 

•  Face  to  face.  Classroom  session  designed  to  synthesize  the  lessons 
learned  from  the  other  three  delivery  methods.  No  longer  lecture- 
based,  these  approaches  typically  involve  interactive  learning  on 
the  complex  areas  (e.g.,  through  practical  exercises,  discussion  of 
case  studies  and  scenarios)  with  direct  coaching  and  mentoring 
from  an  instructor.  This  approach  is  also  used  for  testing  to  ensure 
that  terminal  learning  objectives  were  met.  Such  face-to-face  activ¬ 
ities  accounted  for  20  percent  of  the  training  hours  for  the  course. 

These  four  steps  are  somewhat  analogous  to  the  steps  used  to 
train  military  and  commercial  pilots.  First,  analogous  to  the  use  of 
web-based  IMI,  pilots  attend  ground  school  to  learn  facts  and  concepts 
to  acquire  knowledge  and  comprehension  levels  of  learning.  Second, 
analogous  to  the  use  of  simulations  to  provide  immersion  and  feed¬ 
back,  pilots  learn  to  apply  concepts  with  the  help  of  a  computer-based 
simulation.  Next  they  perform  training  on  an  expensive,  highly  real¬ 
istic  simulator  that  almost  fully  replicates  the  task  of  flying  an  aircraft 
and  gives  the  student  the  opportunity  to  get  full  performance  feedback 
from  the  simulator  and  an  instructor.17  Finally,  pilots  fly  with  the  help 
of  an  instructor,  so  that  students  can  be  tested  and  practice  applications 
on  actual  equipment  and  in  a  fully  realistic  environment. 

Through  the  training  approach  just  described,  IBM  has  been  able 
to  reduce  the  residential  component  of  key  leadership  courses  to  only 
20  percent  of  the  total.  Further,  because  the  company  was  not  so  depen¬ 
dent  on  classrooms  and  travel,  it  was  able  to  dramatically  increase  the 
amount  of  training  provided  without  significant  increases  in  cost. 


17  In  some  respects,  this  is  more  realistic  and  important  than  live  training,  because  students 
can  practice  to  cope  with  dangerous  emergency  situations  that  would  not  be  possible  to  prac¬ 
tice  in  a  live  environment. 
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The  discussion  of  a  commercial  example  in  itself,  even  a  successful 
case  such  as  that  of  IBM,  might  still  be  of  limited  value  to  the  Army, 
given  the  issues  involved  in  translating  an  approach  from  a  commercial 
to  an  Army  context.  However,  as  will  be  seen  in  the  next  subsection, 
we  found  similar  concepts  at  work  inside  the  Army. 

Emerging  DL  Courses  in  the  Army  Also  Show  a  Move  Toward 
Blended  Learning  and  Promising  Benefits 

A  few  courses  in  the  Army  (although  outside  of  TADLP)  are  also 
beginning  to  use  blended  development  and  delivery  methods  in  an 
effort  to  meet  more  of  the  Army’s  needs  for  DL  more  cost-effectively. 
We  will  consider  two  cases  within  NCOES:  the  pilot  BNCOC  (now 
ALC)  Common  Core  (CC)  course  developed  by  TRADOC  with  the 
support  of  the  U.S.  Army  Sergeants  Major  Academy,  and  the  Special 
Forces  (SF)  ANCOC  (now  SLC)  course  developed  by  the  NCO  Acad¬ 
emy  at  the  U.S.  Army  Special  Warfare  Center  and  School. 

Experience  from  these  courses  suggests  that  the  Army  can  effec¬ 
tively  use  multiple  modes  of  DL  and  resident  training  modalities  to 
train  complex  tasks  and  can  convert  a  greater  percentage  of  course  con¬ 
tent  to  DL  than  is  currently  achieved  while,  in  the  process,  signifi¬ 
cantly  reducing  the  time  required  in  residence  courses.  Moreover,  this 
progress  toward  Army  training  goals  in  NCOES  shows  what  can  be 
achieved  by  a  blended  approach  while  reducing  development  costs  and 
the  traditional  inflexibility  of  IMI  course  content  to  change. 

BNCOC  Common  Core.  The  approximately  two-week  BNCOC 
Common  Core  course  has  traditionally  been  available  on  a  residen¬ 
tial  basis,  via  VTT,  or  via  MTT.  Even  though  two  of  these  options 
are  exported  training,  the  choices  were  considered  inadequate  because 
they  failed  to  provide  a  flexible  enough  learning  environment  to  meet 
soldiers’  and  commanders’  needs. 

The  new  version  of  the  BNCOC  Common  Core  will  use  asyn¬ 
chronous  DL  methods  to  create  a  more  flexible  learning  environment  in 
which  students  can  take  a  course  anywhere  and  anytime  over  a  90-day 
period.  Self-paced  IMI,  the  dominant  Army  choice  in  TADLP,  is  one 
of  the  asynchronous  DL  elements  used,  but  it  is  employed  continually 
throughout  the  course  rather  than  in  the  traditional  phased  approach 
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that  puts  all  the  IMI  up  front.  Further,  collaborative  methods  are  used 
to  create  a  virtual  small-group  process  and  to  allow  higher  levels  of 
learning  in  the  DL  portion  than  is  possible  with  IMI  alone. 

The  classes  are  divided  into  learning  modules,  each  containing 
one  or  more  classes  and  several  lessons.  Each  class  contains  some  IMI 
(e.g.,  lectures  with  slides,  videos)  aimed  at  providing  basic  knowledge 
and  completing  ELOs.  TLOs  are  achieved  through  online  discussion 
boards  that  create  an  environment  much  like  the  small-group  environ¬ 
ment  in  residential  training.  Both  student-to-student  and  student-to- 
instructor  interaction  and  collaboration  occur  within  the  process.  The 
discussion  typically  begins  with  a  question  from  the  instructor.  Each 
learner  is  then  assigned  to  post  a  250-word  answer  to  the  discussion 
question.  In  addition,  the  learner  must  post  two  replies  to  their  class¬ 
mates’  responses,  or  post  a  response  to  the  instructor’s  follow-on  ques¬ 
tions.  Instructors  interact  with  the  students  by  providing  feedback  and 
comments  on  practical  exercises  to  ensure  that  the  students  understand 
clearly  and  are  able  to  apply  the  materials  within  the  lessons. 

The  new  BNCOC  Common  Core  course  was  in  a  pilot  stage 
awaiting  validation  at  the  end  of  this  study,  and  thus  its  success  as  a 
DL  approach  had  not  been  established.  Further,  even  if  shown  to  be 
successful,  there  is  a  question  about  whether  the  design  of  the  course 
will  be  easily  transferable  to  other  NCO  courses,  since  by  definition 
the  course  is  a  phase  of  a  larger  BNCOC  (now  ALC)  course  that  covers 
learning  objectives  more  specific  to  particular  occupations.  While  the 
entire  Common  Core  course  is  exported,  it  is  not  clear  to  what  extent 
the  technical,  and  often  more  complex  and  equipment  oriented,  aspects 
of  the  second  (MOS -specific)  phase  of  the  course  could  be  exported. 
However,  it  seems  quite  likely  that  much  of  the  instruction  that  is  done 
in  a  classroom-only  mode  could  be  converted  (e.g.,  if  the  TLO  is  to  be 
able  to  write  or  present  an  operations  order). 

Special  Forces  ANCOC.  Broader  and  already  tested  results  for 
NCO  courses  are  available  in  the  Special  Forces  (SF)  ANCOC  (now 
SLC).  The  general  approach  used  in  the  BNCOC  Common  Core 
was  borrowed  from  the  approach  already  successfully  employed  in 
SF  ANCOC.  That  course  began  as  a  7.5-week  residential  course  that 
included  both  common  and  occupation-specific  elements,  and  it  has 
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been  converted,  validated,  and  successfully  used  for  over  a  year  in  its 
DL  form.18  The  course  intersperses  IMI  and  collaborative  methods  in 
much  the  same  way  as  the  BNCOC  Common  Core  course,  but  also 
has  a  residential  phase.  However,  the  SF  NCO  Academy  has  been  able 
to  reduce  the  residential  portion  of  the  course  to  3  weeks,  thus  achiev¬ 
ing  a  more  than  60  percent  reduction  of  the  course’s  original  length 
of  7.5  weeks.  The  exportable  DL  portion  of  the  course  (nearly  two- 
thirds)  constitutes  a  much  greater  percentage  than  the  15-20  percent 
DL  achieved  by  existing  efforts.19  In  addition,  since  the  course  is  less 
dependent  on  residential  training,  it  is  able  to  offer  more  start  times 
per  year,  thus  allowing  more  learners  to  get  through  the  program  and 
reducing  the  backlog  of  learners  awaiting  the  course. 

These  new  methods  might  potentially  have  other  advantages, 
but  further  study  is  needed.  For  example,  instructor  support  for  the 
new  DL  strategy  will  clearly  be  greater  than  with  the  stand-alone  DL 
method,  but  it  could  well  turn  out  to  be  less  than  that  required  for  resi¬ 
dential  and  YTT  training  methods.  In  the  SF  ANCOC  course,  school 
personnel  estimated  that  the  same  instructor  core  is  needed  to  support 
the  DL  course  as  to  support  all  residential  courses,  but  that  instructor 
core  can  also  accommodate  DL  development  needs  without  additional 
resources.  These  factors  will  have  to  be  verified  with  testing. 


In-House  IMI  Production  Can  Effectively  Support  Blended 
Learning  Courses 

The  IMI  development  methods  used  in  the  BNCOC  and  ANCOC 
blended  learning  courses  described  above  used  instructors  rather  than 
contractors  to  produce  the  content.  There  have  been  several  advantages 


18  Although  the  analysis  is  not  formally  documented,  the  NCO  Academy  has  determined 
that  the  course  is  successful  at  training  the  required  skills  to  an  equal  or  greater  standard 
than  the  residential  course.  One  advantage  of  the  virtual  small-group  process  is  that  all  stu¬ 
dents  participate  equally,  as  opposed  to  the  typical  classroom  situation,  where  there  is  high 
variability  in  how  much  individual  students  participate. 

19  If  a  BNCOC  or  ANCOC  course  is  8  weeks  long,  and  60  hours  is  converted  to  DL  via 
funded  IMI  modules,  the  DL  portion  would  be  less  than  20  percent  (1.5/8). 
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to  this  approach.  First,  IMI  development  was  responsively  integrated 
into  the  total  course  execution  plan.  TADLP  contracting  methods  had 
at  first  been  utilized  to  produce  the  BNCOC  Common  Core  course, 
but  the  courseware  development  process  took  so  long  that  by  the  time 
it  was  complete,  the  courseware  was  obsolete.  As  a  result  of  this  expe¬ 
rience  with  TADLP  processes,  and  the  contrasting  success  of  the  SF 
ANCOC’s  internally  developed  IMI,  the  BNCOC  course  developers 
decided  to  do  the  IMI  development  in  house. 

Second,  organic  development  cost  was  found  to  be  only  a  frac¬ 
tion  of  the  comparable  contracted  course.  In  the  case  of  the  BNCOC 
Common  Core,  the  cost  amounted  to  about  $70,000  to  purchase  the 
authoring  tool  and  pay  TDY  costs  to  instructor/developers.  While  the 
level  of  effort  needed  to  complete  the  development  was  not  recorded, 
the  NCO  Academy  estimates  it  to  be  more  than  a  year  of  effort  but 
substantially  less  than  two  years.  Thus,  total  costs  (including  the  value 
of  the  developer’s  time)  were  likely  somewhere  around  $200,000,  com¬ 
pared  to  a  contracted  cost  that  would  have  ranged  somewhere  between 
$500,000  and  $1  million. 

Another  significant  advantage  of  the  internally  developed  content 
is  that  maintenance  and  updating  of  the  course  can  also  be  completed 
locally,  and  as  often  and  as  soon  as  the  school  thinks  the  changes  are 
needed.  The  ability  to  make  changes  allows  the  instructional  depart¬ 
ments  of  the  school  to  take  full  ownership  of  the  course. 


Chapter  Conclusions 

The  need  to  accommodate  full-spectrum  operations,  modularity,  and 
modernization  has  greatly  increased  the  Army’s  training  and  leader 
development  requirements.  Moreover,  higher  operational  deployment 
rates,  which  are  expected  to  continue  for  the  foreseeable  future,  have 
narrowed  the  windows  when  institutional  training  of  unit  leaders  and 
soldiers  can  be  conducted. 

As  a  result,  the  Army  is  making  major  changes  to  training  and 
leader  development  strategies,  especially  in  its  concepts  for  delivering 
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institutional  training.  A  greatly  expanded  role  for  exported  DL  is  a  key 
element  in  these  emerging  concepts. 

Major  changes  in  the  Army  approach  to  DL  can  lead  to  signifi¬ 
cant  benefits  for  the  Army  and  better  meet  emerging  strategic  objec¬ 
tives.  Emerging  experience  suggests  that  the  Army  can  cost-effectively 
achieve  significantly  greater  exportability  in  DL  content  and  greatly 
increase  the  potential  of  DL  to  meet  expanded  training  requirements. 
While  these  results  are  encouraging,  they  will  need  to  be  verified 
through  ongoing  validations  and  further  tests. 

To  summarize,  we  recommend: 

•  Renewed  focus  and  command  emphasis  on  exportable  DL  for 
leader  development  using  blended  learning  models. 

We  have  concluded  that  such  change  can  lead  to  the  following 
benefits: 

•  Ability  to  significantly  reduce  the  need  for  resident  instruction. 

•  Better  support  of  unit  readiness,  especially  for  units  entering  the 
Train  phase  of  the  ARFORGEN  cycle. 

•  Increased  ability  to  train  critical  tasks  and  skills. 

•  Reduced  development  costs  and  a  greater  return  on  investment. 

•  Expanded  responsiveness  of  DL  to  increasing  demands  for  change. 

•  Increase  in  the  quality  of  DL  content. 

•  Greater  school  buy-in  to  DL  concept. 


CHAPTER  SIX 


Integration  of  TADLP  and  Knowledge 
Management  Learning  Delivery 


The  previous  chapter  discussed  the  first  major  direction  for  broaden¬ 
ing  the  TADLP  beyond  its  current  focus  on  stand-alone  IMI.  In  this 
chapter  we  focus  on  a  second  major  direction,  integration  of  TADLP 
and  knowledge  management  (KM)  learning  delivery.  We  argue  that 
the  Army  should  integrate  KM  learning  delivery  into  the  Army’s  DL 
effort  in  order  to  provide  a  more  comprehensive  and  integrated  DL 
capability. 

We  focus  on  the  function  of  KM  support  of  soldier  and  leader 
learning  in  operational  units  and  call  this  function  “KM  learning  deliv¬ 
ery,”  which  we  define  as  “the  web-based  support  of  individual  soldier 
and  leader  learning  that  takes  place  outside  the  framework  of  formal 
school  courses.”  We  developed  this  definition  to  cover  a  function  we 
consider  to  be  an  inherent  part  of  the  overall  goals  of  a  DL  program 
(provide  learning  anywhere,  anytime),  but  not  currently  under  TADLP, 
which  focuses  on  IMI  support  of  TRADOC  structured  courses. 

The  relationship  between  TADLP  and  KM  learning  delivery  is 
shown  in  Figure  6.1,  which  illustrates  how  the  two  connect  with  the 
ARFORGEN  cycle.  TADLP  is  aimed  more  at  supporting  structured 
courses  that  occur  during  the  Reset  phase,  while  KM  learning  deliv¬ 
ery  programs  provide  support  through  all  ARFORGEN  phases.  The 
two  programs  should  be  integrated  in  a  way  that  provides  the  Army 
with  a  full  DL  capability  to  support  both  structured  and  unstructured 
learning. 
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Figure  6.1 

How  TADLP  and  KM  Learning  Delivery  Complement  Unit  Readiness 
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This  chapter  begins  with  a  description  of  the  KM  program  and  its 
objectives  and  then  discusses  KM  learning  delivery  programs.  We  then 
examine  the  benefits  of  KM  learning  delivery  programs  to  operational 
forces.  We  conclude  the  chapter  by  considering  ways  in  which  TADLP 
and  KM  learning  delivery  programs  could  be  better  integrated. 


The  Knowledge  Management  Program  and  Its  Objectives 

Although  we  focus  our  examination  on  the  distributed  delivery  and  cre¬ 
ation  of  knowledge  in  Army  soldiers  and  leaders,  the  term  KM  refers  to 
a  far  wider  range  of  functions  and  activities.1  The  Army  defines  KM  as 
“the  art  of  creating,  organizing,  applying,  and  transferring  knowledge 


1  The  examination  of  KM  and  KM  learning  delivery  presented  in  this  chapter  was  com¬ 
pleted  at  the  end  of  FY  2008.  Since  that  date,  there  have  been  some  program  changes  and  in 
some  cases  enhancements  or  increased  emphasis.  However,  a  recent  review  showed  that  the 
basic  findings,  conclusions,  and  directions  presented  in  this  chapter  remain  valid. 
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to  facilitate  situational  understanding  and  decision-making.  Knowl¬ 
edge  management  supports  improving  organizational  learning,  inno¬ 
vation,  and  performance.”2 

Expected  Broad  Benefits  from  KM 

AR  25-1,  Army  Knowledge  Management  and  Information  Technology, 
outlines  ambitious  KM  goals:  facilitating  changes  in  governance  and 
culture  to  transform  to  a  knowledge-based  organization;  integrating 
best  business  practices  into  Army  processes  to  promote  a  knowledge- 
based  force;  and  leveraging  the  information  structure  as  an  enterprise 
to  enhance  capabilities  and  efficiencies  so  warfighters  and  business 
stewards  can  act  quickly  and  decisively.3  The  Army  KM  strategy  is 
designed  to  connect  people,  knowledge,  and  technologies  to  achieve 
these  goals;  accordingly,  the  term  KM  applies  to  the  support  of  a  wide 
range  of  activities. 

Operational  KM 

The  Army’s  doctrine  and  modernization  efforts  emphasize  the  use  of 
KM  to  support  the  achievement  of  situational  awareness  and  under¬ 
standing  to  gain  a  major  operational  advantage.  Operational  KM  pro¬ 
vides  a  means  for  soldiers  to  take  advantage  of  favorable  operational 
circumstances  and  to  facilitate  coordinated  and  rapid  action.  To  facili¬ 
tate  effective  operational  KM,  the  Army  is  making  major  investments 
in  C4ISR  systems  and  is  augmenting  unit  authorizations  to  form  KM 
cells  in  operational  units.4  From  this  perspective,  KM  represents  a 
training  requirement,  as  the  Army  must  establish  training  programs  to 
teach  its  soldiers,  leaders,  and  organizations  to  accomplish  KM  tasks 
and  possess  KM  skills,  and  this  will  not  be  a  simple  objective  to  reach. 

KM  Support  of  Training  and  Leader  Development 

The  Army  is  also  anticipating  broad  use  of  KM  to  transform  its  unit, 
individual,  and  leader  training  and  to  align  with  DoD’s  Training 


2  FM  3-0,  Operations,  February  2008. 

3  AR  25-1,  July  2005. 

4  FM  (Interim)  6-0-1,  Knowledge  Management  Cell,  October  2007. 
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Transformation  initiatives.5  The  use  of  KM  to  support  training  is  wide 
ranging  and  includes: 

•  Management  and  execution  of  training  and  leader  development 
programs  in  the  operational,  institutional,  and  self-development 
domains. 

•  Collaborative  development  of  knowledge  (e.g.,  doctrine;  tactics, 
techniques,  and  procedures  (DTTPs))  to  tie  together  the  efforts  of 
the  soldiers  and  leaders  in  institutional  organizations  and  opera¬ 
tional  force. 

•  Delivery  of  training  and  learning  products  to  support  individual 
and  collective  training  and  individual  and  leader  learning  in  oper¬ 
ational  forces. 

For  this  study  we  will  focus  on  this  last  function,  but  we  want  to 
highlight  the  importance  of  the  knowledge  development  function.  The 
key  benefit  of  any  KM  system  is  its  ability  to  foster  and  deliver  relevant 
and  readily  applicable  knowledge.  Due  to  the  rapidly  changing  nature 
of  today’s  full-spectrum  operations,  the  Army  faces  a  significant  chal¬ 
lenge  in  developing  a  complete,  current,  and  valid  knowledge  base  of 
doctrine,  tactics,  techniques,  and  procedures.  From  a  training  and  edu¬ 
cational  perspective,  the  KM  goal  of  facilitating  collaborative  develop¬ 
ment  of  operationally  important  knowledge  is  fundamentally  impor¬ 
tant,  not  only  for  TADLP,  but  also  for  broader  training  and  leader 
development  program  goals. 

Need  for  Substantial  Improvement  in  KM  Programs 

While  the  Army’s  broad  KM  goals  were  established  almost  five  years 
ago,  the  degree  of  change  needed  to  achieve  them  is  large.  Not  surpris- 
ingly,  given  the  complexity  of  the  processes  it  will  support,  the  effort  is 
very  much  in  preliminary  stages. 


5  Secretary  of  the  Army/CSA  Memorandum  (2004).  See  also  Department  of  Defense,  Stra¬ 
tegic  Plan  for  Transforming  DoD  Training,  .February  2009. 
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KM  Learning  Delivery  Programs 

The  Army’s  training  vision  includes  the  establishment  of  an  enhanced 
capability  to  “develop  and  distribute  knowledge  via  a  dynamic,  global 
knowledge  network  called  the  Battle  Command  Knowledge  System 
with  a  purpose  of  providing  immediate  access  to  joint  and  service 
training  and  leader  development  resources.”6 

As  a  result  of  this  direction,  TRADOC’s  commanding  general 
established  a  Battle  Command  Knowledge  System  (BCKS)  organiza¬ 
tion  to  develop  an  Army-level  KM  system  to  support  soldiers  and  lead¬ 
ers  in  the  performance  of  their  operational  missions.  This  organiza¬ 
tion  is  under  the  Combined  Arms  Center  (CAC)  Commander  at  Fort 
Leavenworth,  Kansas,  and  includes  the  U.S.  Army  Forces  Command’s 
(FORSCOM)  Warfighters’  Forums  (WfFs).7 

Three  other  key  Army  programs  directly  support  KM  learning 
delivery  goals:  the  Center  for  Army  Lessons  Learned  (CALL)  and  two 
doctrinal  repositories  on  the  Army  Knowledge  Online  (AKO)  system: 
the  Reimer  Digital  Library  (RDL)  and  “Army  Pubs  and  Forms”  or 
E-Pubs.  Access  to  all  three  of  these  programs  is  available  through 
AKO,  although  access  to  some  sites  or  knowledge  objects  requires  spe¬ 
cial  authorization. 

Figure  6.2  illustrates  the  way  in  which  the  institutional  Army  uses 
these  programs,  not  only  to  support  learning  in  the  operational  force, 
but  also  to  support  its  own  doctrinal  and  training  content  development 
capability. 

These  programs  serve  many  functions: 

•  BCKS  provides  community  forums  for  discussion  and  develop¬ 
ment  of  tacit  knowledge  among  operational  force  soldiers  and 
leaders. 


6  Secretary  of  the  Army/CSA  Memorandum  (2004). 

7  The  BCKS  charter  was  to  provide  technical  and  contract  support  to  assist  the  stand  up  of 
KM  forums.  Once  a  forum  reached  an  initial  operational  capability,  the  management  and 
sustainment  was  turned  over  to  the  sponsoring  organization. 
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Figure  6.2 

KM  Learning  Delivery  Support 


Outputs  Operational  outcomes 
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•  BCKS,  CALL  repositories,  and  CALL  staff  provide  a  point 
for  operational  soldiers  and  leaders  to  access  informal  explicit 
knowledge. 

•  RDL  and  E-Pubs  on  AKO  provide  points  for  operational  sol¬ 
diers  and  leaders  to  access  formal  (e.g.,  Army-approved)  explicit 
knowledge. 

•  The  CALL  and  BCKS  forums,  repositories,  and  staff  provide  an 
important  source  of  input  for  the  development  of  DTTP  and 
course  content  by  the  institutional  Army. 

Note  that  these  capabilities  support  the  institutional  Army  as  well 
as  soldiers  in  the  operational  force.  Importantly,  instructors  and  train¬ 
ing  developers  in  schools  can  and  should  utilize  the  KM  learning  deliv¬ 
ery  forums  and  repositories  to  update  their  courses’  content  and  their 
own  knowledge.  In  addition,  KM  learning  elements  can  be  used  in 
collaborative  phases  of  DL  structured  courses  and  with  guidance  from 
an  instructor.  For  example,  students  in  structured  courses  might  be 
assigned  to  research  important  questions  using  forums  and  repositories. 
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Moreover,  instructors  should  participate  in  the  forums  and  contribute 
to  the  repositories  to  strengthen  their  benefit.  The  role  of  TRADOC’s 
instructional  community  should  include  not  just  course  instruction, 
but  also  the  follow-up  and  updated  learning  accomplished  on  the  KM 
repositories  and  forums. 

BCKS  Program 

The  BCKS  office  has  responsibility  for  developing  and  maintaining 
a  capability  to  support  training,  leader  development  and  education, 
battle  command,  and  doctrine.8  This  office  has  also  been  informally 
designated  as  the  Army’s  proponent  of  operational  KM.9  At  the  end  of 
FY  2008,  the  BCKS  office  had  around  4  military  personnel  and  just 
over  60  DA  civilian  and  contract  personnel.  In  FY  2007,  it  executed 
an  annual  budget  of  just  over  $8  million.10  The  BCKS  office  maintains 
a  website  on  AKO  that  supports  a  collaboration  capability  for  specific 
training  and  leader  development  communities  (forums)  and  also  pro¬ 
vides  a  limited  knowledge  object  repository. 

Forums.  Forums  are  distributed  networks  that  provide  the  ability 
to  link  commanders,  leaders,  staffs,  and  soldiers  from  a  specific  com¬ 
munity  across  the  force  to  establish  virtual  teams  of  professionals  con¬ 
nected  through  online  collaboration  systems.  These  forums  provide  a 
“community  of  practice”  capability  for  peer  discussions  and  profes¬ 
sional  dialogue  among  users  to  enable  shared  learning  and  knowledge 
generation.  For  example,  a  member  of  a  leader  forum  can  ask  other 
forum  members  about  their  ideas  for  counseling  a  subordinate.  With 
the  permission  of  the  forum  leader,  each  forum  is  open  for  any  soldier 
or  civilian  with  an  Army-related  function  and  a  need  to  join. 


8  TRADOC  Regulation  10-5-4,  Organization  and  Functions,  U.S.  Army  Combined  Arms 
Training  Center,  February  2006. 

9  FM  (Interim)  6-0-1,  Knowledge  Management  Cell,  was  authored  by  personnel  in  the 
BCKS  office. 

10  Most  of  this  funding  was  provided  as  a  supplemental  from  year-end  funds.  The  budget  at 
the  beginning  of  the  fiscal  year  was  about  $1.5  million. 
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There  are  over  60  forums  under  the  BCKS  umbrella,  and  this 
number  is  continually  growing.11  The  forums  cover  a  range  of  overlap¬ 
ping  functional,  staff,  leadership,  and  organizational  communities.  A 
sample  listing  of  some  of  these  forums  is  shown  in  Table  6.1. 

We  include  the  Warfighters’  Forums  associated  with  FORSCOM’s 
corps  and  their  subordinate  organizations  as  parts  of  BCKS,  even 
though  they  are  under  the  sponsorship  and  funding  of  FORSCOM 
and  their  responsibilities  include  supporting  mission  command  func¬ 
tions  as  well  as  community-of-practice  functions.12 

Within  each  forum  are  topic  groups,  or  subject  areas  related  to  the 
community,  and  links  to  related  websites.  For  example,  the  Mounted 
Maneuver  Forum  has  topic  groups  like  “Master  Gunners  Section”  and 
links  to  the  Armor  and  Cavalry  Journal  website.  Each  forum  also  has 
its  own  repository  of  key  documents,  articles,  and  other  knowledge 
objects.  Most  forums  send  out  weekly  email  updates  to  forum  mem¬ 
bers  summarizing  important  queries,  recent  articles  of  interest,  and 
other  activities.  Many  of  the  forums  have  a  “Request  for  Information” 
(RFI)  capability.  For  example  on  the  Warfighters’  Forum’s  StrykerNet 
portal  a  user  can  direct  a  question  to  the  forum  staff  who,  supported 
by  the  Corps  and  FORSCOM  staffs,  will  then  research  the  issue  and 
provide  an  answer. 


Table  6.1 

Examples  of  BCKS  Forums 


Community  Area 

Example  Forums 

Functional 

Leadership 

Staff 

Organizational 

Mounted  Maneuver,  Fires,  Civil  Affairs 

NCO,  Company  Commander,  Platoon  Leader,  Command 
S-1,  S3/XO 

Stryker  BCT/I  Corps,  Heavy  BCT/III  Corps,  Army  Reserve 

11  By  “umbrella,”  we  mean  that  these  forums  can  be  accessed  through  the  BCKS  portal. 
However,  most  of  the  forums  can  also  be  accessed  directly  through  AKO. 

12  The  purpose  of  the  Warfighters’  Forums  is  to  “support  training  and  leader  development, 
collaborate  and  share  ideas,  and  find  solutions  to  common  problems.”  Participants  are  offered 
the  opportunity  to  “learn,  innovate,  decide,  and  act  faster  than  our  adversaries  while  oper¬ 
ating  in  a  condition  of  consistently  high  OPTEMPO  in  protracted  conflict.”  FORSCOM 
Warfighters’  Forum  Portal,  https://www.us.army.mil/suite/portal/index.jsp. 
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Each  forum  has  a  forum  sponsor  (e.g.,  a  TRADOC  center  com¬ 
manding  general  or  similar  senior  leader),  a  forum  manager,  and  topic 
leaders.  The  BCKS  office  provides  funding  for  many,  but  not  all,  forum 
managers,  but  the  remainder  of  the  forum  staff  and  contributors  are 
voluntary,  in  that  either  an  organization  or  an  interested  individual 
provides  the  work  “out  of  hide.”  The  forum  leaders  and  topic  manag¬ 
ers  are  the  critical  element  in  successful  forums.  While  respecting  the 
need  for  open  discussion,  forum  leaders  and  topic  managers  must  pro¬ 
vide  a  certain  amount  of  policing,  and  must  aggressively  search  for  rel¬ 
evant  knowledge  objectives  to  post  and  distribute  to  forum  members  to 
keep  the  forums  current.  They  are  also  responsible  for  seeing  to  it  that 
knowledge  is  shared  more  generally  when  appropriate. 

The  content  of  the  forums  varies  by  the  level  of  leader  and  topic 
manager  effort  and  sponsor-provided  resources.  One  of  the  more  well- 
supported  and  expansive  forums  is  the  Stryker  Brigade  Combat  Team 
Warfighters’  Forum  (SWfF).13  The  SWfF  is  a  community  of  purpose 
within  the  Stryker  Brigade  Combat  Team  FORSCOM  and  TRADOC 
community,  with  its  StrykerNet  WfF  portal  providing  a  conduit  for 
operational  experience  to  enable  quick  adaptation  and  high  perfor¬ 
mance.14  The  SWfF  has  a  ten-person  permanent  staff  and  a  robust 
capability  to  develop  content  and  provide  tailored  responses  to  requests 
for  information.  It  includes  a  number  of  features  demonstrating  that 
BCKS  and  its  forums  could  complement  and  extend  the  Army’s  DF 
capabilities.  These  include  leadership  and  operational  training  classes, 
exercises,  and  learning  tools. 

Repositories.  To  supplement  forum  repositories,  the  Warrior 
Knowledge  Base  (WKB)  provides  a  web-based,  central  BCKS  reposi¬ 
tory  of  data,  information,  references,  and  knowledge  (objects)  needed 
by  BCKS  users.  It  is  focused  on  achieving  information  interoperabil¬ 
ity  across  BCKS  objects.  Besides  providing  an  internal  repository,  the 
WKB  also  has  links  to  the  other  BCKS  forums,  CAFF,  and  the  AKO’s 
RDF  and  E-Pubs  site. 


13  Hallmark  and  Gayton  (forthcoming)  for  a  detailed  description  and  assessment  of  this 
forum’s  potential. 

14  StrykerNet  Staff  (2010). 
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In  addition  to  the  BCKS  staff,  11  soldiers  from  the  USAR’s  84th 
Division  support  the  WKB  effort  on  weekends  and  during  a  two-week 
annual  training  period.  They  review  documents,  determine  which 
should  be  included  in  the  WKB  repository,  and  provide  searchable 
keywords  (also  called  “metadata”)  to  support  search  functions.  This 
repository  is  established  by  its  manager  and  contains  objects  submitted 
by  forum  leaders.  While  labeled  a  BCKS  repository,  WKB  does  not 
currently  include  a  capability  to  directly  search  within  other  forums. 
The  lack  of  such  a  search  capability  is  partly  the  result  of  the  need  to 
protect  forum  member  internal  discussions,  but  it  is  also  due  to  the  dif¬ 
ficulty  of  developing  the  technical  capability  to  search  across  forums. 

CALL  Program 

CALL  is  a  TRADOC  Center  operating  under  the  CAC  commanding 
general.  CALL  “collects,  analyzes,  disseminates,  and  integrates  new 
concepts,  TTR  and  solutions  throughout  the  Army  from  the  tactical 
through  strategic  levels  of  war,  with  a  primary  focus  on  operational 
issues,  as  an  agent  of  change  throughout  the  Army.”15  At  the  end  of 
2008,  CALL  had  approximately  12  military  and  about  170  DA  civil¬ 
ian  and  contract  personnel;  its  annual  budget  was  just  over  $17  million 
in  FY  2007. 

Compared  to  the  BCKS  forums,  CALL  has  a  fairly  robust  staff  of 
SMEs  who  work  full  time  at  Fort  Leavenworth,  key  TRADOC  centers 
and  schools,  Combat  Training  Centers  (CTCs),  and  with  units.  This 
staff  researches  “observations,  insights,  and  lessons”  (OIL),  assesses 
their  value,  and  develops  products  from  them  that  provide  vetted  “les¬ 
sons  learned”  to  the  Army.  These  are  distributed  in  hard  copy  and  in 
online  documents  that  range  from  single-page  “Lessons  of  the  Day”  to 
pamphlets  of  100  or  more  pages  covering  emerging  TTP  on  an  area  of 
operational  importance,  such  as  TTP  for  stability  and  support  opera¬ 
tions.  CALL  also  produces  lessons  learned  videos,  which  are  available 
both  online  and  through  distribution. 


15  CALL’s  mission  statement,  from  U.S.  Army  Combined  Arms  Training  Center,  Organiza- 
tion  and  Functions,  TRADOC  Regulation  10-5-4,  February  2006. 
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CALL’s  KM  learning  delivery  is  made  up  of  two  useful  functions. 
First,  CALL  maintains  open,  restricted,  U.S.  classified,  and  NATO 
classified  websites,  which  provide  a  repository  for  all  CALL-developed 
products  as  well  as  related  documents  that  its  analysts  have  found  of 
potential  benefit  to  units  and  other  users.  These  sites  use  topic  tabs  (e.g., 
branch,  echelon,  training),  which  allow  users  to  quickly  find  products 
of  interest.  There  are  also  several  types  of  search  capabilities,  a  cata¬ 
logue  of  products,  and  links  to  related  sites. 

A  second  useful  function  is  CALL’s  RFI  capability,  which  is  avail¬ 
able  to  U.S.  military  and  government  agencies,  coalition  partners,  and 
civilians  with  an  approved  need  for  access.  Military  analysts  search 
the  archives  and  contact  SMEs  to  provide  the  information  back  to 
the  requestor.  The  goal  is  to  provide  a  response  within  24  hours  for 
deployed  units  and  72  hours  otherwise.  In  FY  2008,  CALL  responded 
to  an  average  of  over  900  RFIs  a  month.  Obviously,  the  level  of  effort 
that  can  be  provided  is  limited  by  the  size  of  CALL’s  staff,  but  at  a 
minimum  it  is  able  to  send  a  package  of  relevant  material.  In  cases  of 
greater  urgency  or  importance,  a  more  extensive  “staff  study”  effort  can 
be  made  to  access  sources  outside  of  the  CALL  archives  and  provide  a 
tailored,  more  extensive  response. 

As  an  important  supporting  capability,  CALL  has  representatives 
at  key  points  throughout  the  Army  to  support  its  collection  and  dis¬ 
semination  mission.  For  example,  CALL  representatives  are  located  at 
each  division  and  corps,  the  CTCs,  mobilization  sites,  and  TRADOC 
schools  and  centers.  This  wide  distribution  of  CALL  personnel  has 
proved  valuable  because  it  not  only  provides  an  “inside  ear”  to  hear 
about  ongoing  operations  and  initiatives,  but  also  facilitates  the  dis¬ 
semination  of  information  to  help  unit  leaders  conduct  their  opera¬ 
tional  and  training  missions. 

CALL  also  provides  MTT  support  on  request.  The  MTT  sup¬ 
ports  deploying  and  training  units  by  providing  a  snapshot  of  the 
current  operational  environment,  which  is  based  on  near-real-time 
in-theater  observations,  lessons  learned,  AARs,  and  TTPs.  It  is 
designed  as  a  seminar  with  information  briefings  and  facilitated  dis¬ 
cussion,  not  as  a  tactical-level  training  tool. 
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CALL’s  primary  limitation  is  simply  the  size  of  its  staff.  Because 
its  staff  resources  are  limited  and  response  times  are  short  by  design, 
the  knowledge  objects  it  addresses  generally  cover  a  narrow  area  and 
focus  on  specific  situations  rather  than  the  comprehensive  coverage 
generally  presented  in  doctrinal  publications.  In  addition,  as  with 
all  KM  repositories,  search  capabilities  pose  a  problem  for  the  time- 
constrained  unit  user. 

Army  Knowledge  Online  Doctrinal  Repositories 

The  Reimer  Digital  Library  (RDL)  and  E-Pubs  are  repositories  on 
AKO  that  contain  all  approved  Army  publications,  including  doctrinal 
Field  Manuals,  Training  Support  Products,  Soldier  Training  Publica¬ 
tions,  regulations,  pamphlets,  and  other  such  official  publications.  The 
repositories  enable  the  user  to  look  up  publications  by  type  and  by  series 
or  number.  A  search  of  publications  is  possible  if  the  searcher  generally 
knows  what  type  of  publication  to  look  for,  but  a  search  capability 
focusing  on  specific  content  in  a  publication  or  across  publications  is 
not  supported.  AKO  also  has  a  search  engine,  but  its  utility  is  limited. 

A  major  issue  with  the  content  of  these  AKO  repositories  is  cur¬ 
rency  of  these  authoritative  references.  The  Army  has  been  challenged 
to  develop  a  complete  set  of  doctrine  and  fully  vetted  TTPs  for  the  cur¬ 
rent  operational  requirements.  For  example,  the  Army’s  operational- 
level  doctrine  for  counterinsurgency  operations  was  published  in  late 
2006,  over  three  or  more  years  after  these  types  of  operations  began  in 
Afghanistan  and  Iraq.16  Moreover,  the  tactical-level  doctrine  for  coun¬ 
terinsurgencies  is  still  under  development  and  has  not  yet  been  codified 
into  a  published  FM. 

The  major  cause  of  this  issue  is  that  the  operational  pace  and  the 
pace  at  which  enemies  adapt  has  combined  to  increase  the  rate  at  which 
doctrinal  and  training  products  must  be  revised,  enhanced,  and  aug¬ 
mented.  It  is  thus  difficult  for  the  institutional  Army  to  keep  current 
with  operational  requirements  and  practices.  A  contributing  factor  is 
the  general  reduction  of  TRADOC  staffing  over  the  past  decade  or 
more. 


16 


FM  3-24,  Counterinsurgency,  December  2006. 
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A  major  goal  of  the  KM  program  is  to  provide  for  more  effec¬ 
tive  collaboration  between  the  operational  and  institutional  Army  to 
increase  the  rate  at  which  enhanced  doctrinal  and  training  content  can 
be  developed  as  well  as  delivered.  In  this  respect,  CALL  and  BCKS 
KM  capabilities  represent  as  important  a  resource  for  the  institutional 
Army  as  they  do  for  soldiers  and  leaders  in  the  operational  force. 

There  are  many  other  websites  on  AKO  with  information  and 
knowledge  products  that  can  promote  learning  in  soldiers  and  leaders 
in  operational  units.  For  example,  each  TRADOC  proponent  school 
has  a  home  page,  and  often  the  latest  drafts  of  doctrinal  materials  are 
available  on  these  sites.  Similarly,  most  units  and  organizations  have 
websites  with  relevant  materials.  However,  there  is  currently  no  truly 
usable  search  capability  among  the  various  websites.  This  lack  is  partly 
due  to  the  fact  that,  for  security  reasons,  many  of  the  sites  and  subsites 
require  authorization  from  the  site  or  subsite  manager  to  gain  access. 

Importantly  for  this  study,  the  IMI  modules  developed  and  being 
developed  under  TADLP,  and  which  certainly  would  be  helpful  given 
these  objectives,  are  not  directly  accessible  through  AKO  unless  the 
user  is  enrolled  in  the  course  supported  by  the  module. 


Ongoing  KM  Learning  Delivery  Enhancements 

The  high-level  interest  in  improving  overall  KM  capabilities,  including 
KM  learning  delivery,  is  evidenced  by  the  emphasis  the  senior  Army 
leadership  has  expressed  in  a  series  of  KM  guidance  documents.  For 
example,  a  July  2008  memorandum  from  the  Secretary  and  Chief  of 
Staff  of  the  Army  stressed  Army  KM  principles  and  emphasized  the 
importance  of  previous  Army  KM  memoranda  and  AR  25-1,  Army 
Knowledge  Management  and  Information  Technology A 

The  emphasis  by  TRADOC  leadership  seems  equally  strong. 
General  Wallace,  a  former  TRADOC  commanding  general,  commis¬ 
sioned  a  study  to  examine  and  make  recommendations  for  improv¬ 
ing  TRADOC’s  KM  programs.  This  extensive,  well-structured  study 


17  See  HQDA  (July  2008). 
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was  conducted  by  BAE  Systems  and  concluded  that  TRADOC’s  KM 
programs  needed  greater  synchronization  and  improvement  across  the 
board.18 

Partly  as  a  result  of  the  BAE  Systems  study,  in  January  2008  the 
TRADOC  commander  directed  the  formation  of  the  Combined  Arms 
Center-Knowledge  (CAC-K)  organization  to  address  the  challenges 
outlined  in  the  BAE  Systems  report.19  CAC-K  is  to  be  “an  organiza¬ 
tion  that  synchronizes  knowledge  creation,  organization,  application, 
and  transfer  in  order  to  enable  understanding  and  support  decision 
making.”  This  is  a  positive  development  in  terms  of  KM  improvement, 
especially  in  terms  of  achieving  greater  unity  of  effort  by  bringing 
together,  under  a  single  organization,  not  only  CALL  and  the  BCKS 
office,  but  also  CAC’s  Combined  Arms  Doctrine  Directorate  (CADD), 
Combat  Studies  Institute  (CSI),  and  Military  Review,  all  of  whom  play 
a  role  in  Army  learning  delivery.20 


Benefits  of  KM  Learning  Delivery  Programs  to 
Operational  Forces 

To  assess  the  benefits  of  the  KM  learning  delivery  programs,  we  exam¬ 
ined  three  primary  sources:  our  own  recently  conducted  study  of  leader 
development  in  units,  the  2006  Army  Research  Institute  (ARI)  Sample 
Survey  of  Military  Personnel,  and  the  BAE  Systems  study  of  KM.  We 


18  BAE  Systems  (2007). 

19  Headquarters  Combined  Arms  Center,  OPORD  08-050A,  Formation  of  CAC-Knowledge, 
February  2008. 

20  The  role  of  CADD  is  described  in  TRADOC  Regulation  10-5-4,  Organization  and  Func¬ 
tions:  CADD  performs  “overall  management,  integration,  and  quality  control  of  Army  doc¬ 
trine;  integrates  Army  doctrine  vertically  and  horizontally  within  the  Army;  integrates  Army 
doctrine  with  joint  and  multinational  doctrine;  represents  the  Army  in  multinational  doc¬ 
trine  committees;  and  performs  other  doctrine-related  tasks.”  TR  10-5-4  outlines  CSI’s  mis¬ 
sion  as  follows:  “Conduct  and  publish  research  on  historical  topics  pertinent  to  the  current 
doctrinal  concerns  of  the  Army.”  The  Military  Review  mission  is  to  “create  a  long-term  sched¬ 
ule  of  themes  and  develop  articles  which  facilitate  full  discussion  of  emerging  issues  to  assist 
the  development  of  concepts  and  doctrine,”  and  to  “establish  a  network  with  other  Services, 
other  Army  professional  journals,  and  other  agencies  to  improve  mutual  support.” 
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also  visited  selected  KM  learning  delivery  sites  in  order  to  obtain  an 
independent  estimate  of  their  benefit  from  a  unit  leader  perspective. 

RAND  Study  of  Leader  Development  in  Units 

In  2006  RAND  conducted  a  study  of  leader  development  in  opera¬ 
tional  units,  which  included  an  extensive  set  of  surveys  of  and  inter¬ 
views  with  405  officers  from  the  grade  of  lieutenant  to  major.21  The 
survey  asked  officers  whether  they  had  participated  in  an  online  forum 
(e.g.,  BCKS).  It  also  provided  a  list  of  12  leader  development  activities 
and  asked  them  to  select  the  top  three  activities  that  “were  most  effec¬ 
tive  in  developing  your  leadership  qualities  and  teaching  leadership  les¬ 
sons.”  These  activities  included  “participation  in  online  forums”  and 
also  such  activities  as  professional  reading  and  participation  in  train¬ 
ing  events  such  as  a  Command  Post  Exercise  (CPX)  or  Field  Training 
Exercise  (FTX). 

The  surveys  found  that  42  percent  of  the  majors  and  senior  cap¬ 
tains  and  36  percent  of  the  junior  captains  and  lieutenants  participated 
in  online  forums.  In  terms  of  leadership  development  benefits,  only 
7  of  282  members  of  the  junior  group  and  2  of  123  members  of  the 
senior  group  selected  “online  forums”  as  one  of  the  top  three  most 
effective  activities,  and  this  was  the  least-selected  activity  overall. 

This  low  ranking  should  not  be  viewed  as  a  criticism  of  the  ben¬ 
efit  of  these  forums.  Many  of  the  other  choices  involved  activities  that 
more  directly  involved  exercise  of  leadership  skills  (e.g.,  deployments, 
CTC  and  training  events).  Considering  this,  the  fact  that  the  forums 
were  chosen  as  one  of  the  top  three  activities  by  anyone  at  all  actually 
indicates  a  reasonable  perceived  benefit. 

A  finding  of  greater  concern  is  that  so  few  officers  in  the  survey 
group  had  even  used  the  forums.  Much  of  our  concern  stems  from 
the  fact  that  the  pioneering  forum,  and  probably  the  best  known,  is 
companycommander.com,  and  this  one  would  be  well  aimed  at  this 
group. 


21  Schirmer  et  al.  (2008).  The  study  surveyed  and  interviewed  282  junior  captains  and  lieu¬ 
tenants  and  123  senior  captains  and  majors. 
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Our  follow-up  interviews  provided  some  insights  on  the  bene¬ 
fits  of  the  forums  and  repositories  and  how  they  could  be  improved. 
In  general,  those  who  used  these  forums  found  them  at  least  some¬ 
what  useful.  A  few  participants  found  them  highly  useful,  but  most 
saw  them  as  one  of  the  sources  of  information  they  might  use  to  sup¬ 
port  immediate  operational  or  training  needs,  and  less  so  as  sources  of 
advice  on  good  leadership  techniques  and  traits,  which  was  the  focus  of 
the  survey  questions.  When  this  topic  was  discussed,  a  general  thread 
was  that  “There  is  too  much  information  out  there”  and  that  there 
is  great  difficulty  in  quickly  finding  the  specific  information  needed. 
Almost  everyone  surveyed  seemed  to  think  that  the  current  capability 
to  do  this  is  inadequate.  The  inadequacy  of  effective  search  engines  on 
the  forums  and  for  AKO  itself  was  nearly  always  brought  up  during 
discussions  of  this  topic. 

There  also  seemed  to  be  limited  confidence  in  the  benefit  of 
some  of  the  information  available  through  the  forums  and  reposito¬ 
ries.  Many  said  that  “You  don’t  know  how  good  it  is.”  Such  comments 
applied  to  both  BCKS  and  CALL.  Most  officers  seemed  to  prefer  going 
to  someone  they  knew,  whose  views  they  felt  they  could  trust,  rather 
than  accepting  outright  the  advice  or  information  received  through  the 
forums.  This  does  not  mean  that  the  forums  were  not  used — the  data 
indicate  otherwise — but  that  most  saw  them  as  supplementary  infor¬ 
mation  sources. 

ARI  Distance  Learning  Findings 

During  late  2006,  ARI  conducted  a  survey  of  just  under  4,800  officers 
and  just  over  5,200  soldiers  and  NCOs  on  a  range  of  topics,  including 
the  frequency  of  their  use  of  AKO  forums  and  communities  and  par¬ 
ticipation  in  online  discussion  groups.22  The  results  showed  that  over 
80  percent  of  officers  used  AKO  forums  or  communities  “not  at  all”  or 
“infrequently.”  For  NCOs,  the  comparable  rate  was  around  70  percent. 


22  ARI,  Distance  Learning  Findings  from  the  Fall  2006  Sample  Survey  of  Military  Personnel, 
March  27,  2007. 
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RAND  Stryker  Warfighters'  Forum  (SWfF)  Study 

In  2009  RAND  conducted  an  assessment  of  the  SWfF.23  This  study 
included  a  web-based  survey  that  examined  the  StrykerNet  forum 
usage.  The  survey  had  1,359  officer  and  NCO  respondents,  and  the 
results  showed  that  only  about  one  in  four  respondents  had  visited  the 
site  and  over  half  of  those  used  it  for  training  or  self-development. 

While  these  usage  rates  are  similar  to  those  described  above,  the 
study  pointed  out  that  the  benefits  of  such  a  forum  are  not  limited 
to  direct  usage,  but  also  include  indirect  benefits.  For  example,  if  a 
unit  leader  used  the  site  to  support  a  leader  development  session  or 
to  revise  the  unit’s  tactical  standard  operating  procedures  (SOPs),  or 
if  a  doctrine  developer  used  the  forum  as  a  source  for  revising  a  field 
manual,  the  benefit  would  reach  a  far  wider  audience  than  individuals 
who  directly  used  the  site. 

BAE  Systems  KM  Study 

As  a  part  of  the  BAE  Systems  KM  study  effort,  over  300  individuals  in 
units  and  TRADOC  were  interviewed,  and  surveys  were  conducted  of 
over  4,000  NCOs  and  officers.24  The  study  found  that 

•  “Much  of  the  perception  from  the  field  is  that  DTTPs  (Doctrine, 
Tactics,  Techniques,  and  Procedures)  are  often  not  current,  timely 
or  relevant.” 

•  “The  primary  factor  that  would  encourage  Soldiers  to  contribute 
more  OIL  (Observations,  Insights,  and  Lessons)  is  increased  con¬ 
fidence  in  the  vetting  process.” 

•  “No  systematic  process  exists  to  quickly  and  efficiently  extract 
“knowledge  nuggets”  from  diverse  sources  of  content.” 

Overall,  this  study,  while  confirming  the  need  for  an  effective  KM  pro¬ 
gram,  concluded  that  improvements  were  necessary  to  better  achieve 
the  program’s  need  and  potential. 


23  Hallmark  and  Gayton  (forthcoming).  This  report  also  provides  insights  into  why  soldiers 
used  the  SWfF  and  demonstrated  its  use  to  develop  and  refine  training  content. 

BAE  Systems  (2007). 
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Conclusions,  Directions  for  Improvement,  and  Better 
Integration  with  TADLP 

We  have  concluded  that  the  functions  provided  by  KM  learning  deliv¬ 
ery  are  highly  important  to  leader  training  and  development.  They 
serve  a  key  DL  function  and  complement  TADLP’s  effort  to  support 
structured  learning  in  the  framework  of  formal  courses.  The  learning 
that  a  soldier  gets  in  the  institution  can  decay  after  the  completion  of 
the  course,  elements  of  it  can  become  dated,  and  new,  important  ele¬ 
ments  may  be  added  to  the  course.  Thus,  KM  learning  delivery  forums 
and  repositories  provide  an  updating  and  refreshing  function  that  aug¬ 
ments  formal  institutional  training. 

The  capabilities  provided  by  the  Army’s  major  KM  learning  deliv¬ 
ery  programs  provide  a  range  of  knowledge  support  to  the  operational 
force,  as  summarized  in  Table  6.2. 


Table  6.2 

KM  Learning  Delivery  Capabilities 


KM  Learning 

Delivery  Program  Contributions  Limitations 


BCKS 


CALL 


AKO  RDL/ 
E-Pubs 


•  Supports  development  of  tacit 
knowledge  via  collaborative 
forums 

•  Rapidly  distributes  relevant 
knowledge  objects  (e.g.,  journal 
articles)  directly  to  community 
members 

•  Maintains  repository  of  current, 
relevant  knowledge  objects 

•  Has  repository  of  vetted,  current 
TTP 

•  RFIs  provide  responsive, 
researched  answers  to  queries 
from  unit  leaders 

•  Unit  representatives  can  help  units 
leaders  use  CALL  capabilities 

•  Provides  fully  vetted  doctrine  and 
TTP 


Limited,  but  some  vetting 
Limited  search  capability 
Stovepipe  nature  of  forums 
Staff  size 


Knowledge  objects 
compartmentalized 
RFI  responses  limited  by 
staff  size/expertise/time 
Search  capability  limited 
Staff  size 


Much  content  is  not  current 
and  many  key  areas  not 
currently  covered 
No  effective  search 
capability 
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BCKS  provides  the  most  responsive  support  and  contributes  to 
tacit  knowledge  development;  CALL  provides  responsive,  somewhat 
vetted  TTP;  RDL  provides  fully  vetted  doctrine  and  TTP  but,  as 
pointed  out  previously,  TRADOC’s  processes  and  doctrinal  develop¬ 
ment  resources  have  been  greatly  challenged  by  the  amount  and  rate  of 
change  in  the  operational  environment  and  the  ensuing  requirements 
for  training  support  material.  A  challenge  for  users  is  the  lack  of  inte¬ 
gration  across  sources,  i.e.,  a  user  must  perform  multiple  searches  to  get 
desired  information. 

Needed  Improvements 

Based  on  our  review,  it  is  fair  to  say  that  while  these  efforts  are  seen 
as  useful  by  many  leaders  in  the  operational  force,  a  far  larger  number 
either  do  not  use  them  or  see  them  as  being  of  limited  benefit.  The  lack 
of  a  significant  benefit  is  related  to  a  combination  of  factors,  includ¬ 
ing  the  limited  time  unit  leaders  have  available  for  such  efforts  and  the 
difficulty  of  searching  for  effective  solutions  among  the  vast  number 
of  sites  and  knowledge  objects  available.  As  a  whole,  unit  leaders  want 
access  to  reasonably  good  solutions  to  their  specific  knowledge  needs 
quickly.  Since  there  is  no  simple  or  inexpensive  path  to  fully  or  even 
partially  provide  this  capability,  those  responsible  for  making  improve¬ 
ments  in  this  area  face  real  challenges. 

A  more  responsive  ability  to  update  and  disseminate  doctrine  and 
TTP  and  to  supplement  the  structured  education  (formal  courses)  of 
leaders  are  both  acknowledged  areas  for  improvement.  Moreover,  we 
found  that  the  organizations  with  the  responsibility  for  performing 
these  functions  are  making  reasonable  efforts  to  improve  them. 

Currently,  there  are  ongoing  efforts  to  develop  IT  support  capa¬ 
bilities  and  to  achieve  better  synchronization  of  existing  KM  learning 
delivery  efforts.  While  such  efforts  are  likely  to  be  helpful,  there  is  little 
reason  to  think  that  major  improvements  are  likely  in  the  near  or  even 
middle  term.  The  need  for  improvement  is  great  and  there  appear  to  be 
no  major  increases  likely  in  program  resources.25 


25  Based  on  HQDA,  “TSPU  MDEP:  Training  Support  to  Units,”  March  19,  2008. 
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At  the  heart  of  the  matter  is  the  “people”  part  of  KM.  The  effec¬ 
tiveness  of  the  processes  used  to  develop  content  and  its  effective  dis¬ 
tribution  will  depend  primarily  on  the  amount  of  expert  manpower 
involved.  For  example,  to  support  an  effective  search  capability,  the 
value  of  the  content  needs  to  be  assessed,  and  the  content  needs  to  be 
broken  into  discrete  knowledge  objects  and  meta-tagged  with  descrip¬ 
tions  to  facilitate  search.  The  effort  needed  to  do  this  across  all  KM 
repositories  is  large.  There  are  promising  technologies  that  could  sup¬ 
port  such  an  effort,  but  in  the  end,  expert  judgment  will  be  the  key 
resource  needed  for  significant  improvement.  Even  with  better  meta¬ 
tagging  and  more  effective  search  engines,  expert  manpower  will  still 
be  needed  to  take  the  wide  range  of  inputs  and  coalesce  them  into 
packages  suitable  for  most  unit  users. 

The  major  limitation  that  could  impede  these  initiatives  is  staff 
size,  especially  for  BCKS.  Much  of  the  staffing  is  provided  by  supple¬ 
mental  and  end-of-year  funding.  In  addition,  these  missions  are  the 
responsibility  of  specialized  organizations  that  are,  in  most  aspects, 
outside  the  mainstream  TRADOC  instructional  and  doctrinal  devel¬ 
opment  effort. 

Directions  for  Improvement 

The  KM  learning  delivery  mission  is  highly  important  in  light  of  the 
current  and  expected  future  operational  environment.  While  many 
organizations  are  supporting  this  effort,  and  great  benefit  to  the  opera¬ 
tions  force  has  been  provided,  a  more  integrated  effort  seems  clearly 
to  have  the  potential  to  provide  great  benefits  to  the  operational  force. 
Unit  users  do  not  have  the  time  to  search  through  all  the  potential 
sources  of  information  to  find  answers  and  solutions.  They  gener¬ 
ally  want  a  concise  “good  enough”  informational  product  that  can  be 
directly  adapted  to  their  specific  situation,  rather  than  a  set  of  refer¬ 
ences  that  needs  lengthy  review  and  evaluation. 

Enhanced  RFI  Capability.  A  single  supported  RFI  capability  could 
be  created  that  would  include  a  capacity  to  go  to  the  “TRADOC 
expert”  to  formulate  a  response.  TRADOC’s  proponent  system  could 
be  used  to  identify  topic  leads,  who  would  serve  as  a  school’s  experts 
in  a  selected  area  and  who  would  be  responsible  for  keeping  current  in 
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the  topic  area.  For  example,  the  MP  School  could  designate  an  expert 
for  the  topic  “preserving  the  integrity  of  an  incident  site.”  The  overall 
concept  would  be  to  create  a  “telephone  book”  of  experts  on  a  long  list 
of  specific  tasks  and  topics. 

Forum  Integration.  Some  combination  of  greater  integration  and 
streamlining  of  the  BCKS  forums  also  seems  to  be  needed.  While 
there  is  obvious  value  in  the  concept  of  small,  informal  communities, 
it  is  hard  to  understand  why  there  are  separate  forums  for  Heavy  BCTs 
and  Mounted  Maneuver.  A  leader  looking  for  a  suggestion  from  the 
armored/mechanized  forces  community  should  not  have  to  make  sepa¬ 
rate  requests  to  each  forum.  A  BCKS  weekly  update  system  could  be 
integrated  so  that  each  soldier  receives  a  single,  tailored  update. 

Improved  Search  Capability.  The  need  for  an  effective  search  capa¬ 
bility  across  KM  learning  delivery  sites  has  been  a  well-documented 
and  recurring  theme  in  this  chapter.  Across  TRADOC’s  KM  learning 
delivery  program  staffs,  we  also  found  full  awareness  of  the  need  to 
enhance  capabilities  to  search  within  and  among  the  various  reposito¬ 
ries;  these  staffs  are  making  major  efforts  to  improve  technical  search 
capabilities.  But  the  challenges  of  creating  effective  search  capabilities 
are  significant,  especially  given  security  needs  and  limited  access  to  spe¬ 
cific  sites.  There  has  been  considerable  effort  in  this  area,  to  include  con¬ 
sideration  of  commercial  solutions  such  as  Google,  but  limited  progress. 
The  solution  cannot  be  based  on  IT  alone.  The  capacity  must  be  based 
on  effective  meta-tagging,  which  requires  the  judgment  of  many  subject 
matter  experts  and  a  common  system  across  sites. 

KM  Learning  Delivery  as  a  Core  TRADOC  Mission.  Overall,  we 
recommend  that  a  fully  effective  KM  learning  delivery  capability 
become  a  core  TRADOC  mission,  and  that  this  effort  should  be  sys¬ 
tematically  supported  by  the  integration  of  all  the  expertise  residing  in 
TRADOC. 

Integration  of  TADLP  and  KM  Learning  Delivery 

We  conclude,  further,  that  the  efforts  of  KM  learning  delivery  pro¬ 
grams  and  TADLP  should  be  better  integrated  for  the  mutual  ben¬ 
efit  of  both,  and  to  achieve  an  enhanced  leader  development  capability 
overall.  One  possible  integration  alternative  would  be  to  combine  these 
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programs.  However,  the  focus  areas  of  KM  and  TADLP  differ,  and 
combining  them  into  a  single  program  would,  in  our  estimation,  create 
an  unmanageable  scope  of  effort.  For  example,  the  BCKS  forums  and 
CALL  efforts  involve  not  only  learning  delivery,  but  also  collection 
and  analysis  of  knowledge.  The  KM  programs  have  been  shaped  to 
accomplish  all  three  functions  and  should  not  be  separated  out.  Thus, 
while  we  do  not  recommend  combining  the  two  programs,  we  do  rec¬ 
ommend  that  one  organization  should  take  the  lead  in  integrating  and 
advocating  for  efforts  to  support  learning  in  the  operational  force.  The 
TCM  for  TADLP  is  the  logical  choice,  given  the  TCM’s  charter  of 
coordinating  the  Army’s  overall  distance  learning  efforts. 

The  major  need  is  to  establish  a  more  collaborative,  mutually  sup¬ 
porting  effort  between  the  schools’  instructors  and  training  and  doc¬ 
trine  developers  on  the  one  hand,  and  the  CALL  and  BCKS  programs 
on  the  other.  TRADOC’s  instructor  pool  represents  an  important  part 
of  the  Army’s  institutional  knowledge,  which  should  be  shared  and 
available  outside  the  scope  of  structured  instruction.  Also,  given  the 
rate  at  which  operational  experience  becomes  dated,  the  KM  sites  pro¬ 
vide  an  important  mechanism  for  maintaining  instructor  currency. 

Obviously,  school  SMEs  can,  and  many  do,  benefit  from  the 
updated  capability  of  CALL  and  BCKS.  Besides  using  their  capabili¬ 
ties,  school  SMEs  should  support  CALL  and  BCKS  efforts  by  par¬ 
ticipation.  This  appears  to  be  happening  in  many  cases  now,  as  the 
CALL  and  BCKS  sites  are  used  and  supported  by  many  school  SMEs, 
but  such  participation  is  generally  on  an  individual  basis;  it  could  be 
encouraged  and  perhaps  better  coordinated  by  policy  and  command 
guidance. 

Another  example  illustrating  how  integration  between  CALL  and 
BCKS  could  be  of  benefit  would  be  the  placement  of  IMI  modules 
that  present  knowledge  ELOs  on  the  BCKS  and  CALL  sites  and  make 
them  findable  using  KM  search  systems.  Updated  content  should  be 
available  to  all  soldiers,  not  just  those  enrolled  in  a  course. 

Moreover,  as  course  material  is  updated,  there  should  be  a  mecha¬ 
nism  to  identify  critical  changes  and  to  disseminate  them  through  an 
integrated  forum  system.  For  example,  if  the  infantry  school  devel¬ 
ops  an  enhanced  technique  for  searching  a  room,  and  develops  a 
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video  showing  and  explaining  the  technique,  that  product  could  be 
announced  on  the  forum  weekly  updates,  and  placed  in  CALL  and 
BCKS  repositories.  Besides  providing  benefit  to  the  operational  force, 
this  dissemination  would  provide  a  mechanism  for  school  instructors 
and  training  developers  to  get  feedback  from  a  broad  range  of  leaders 
in  the  field,  thus  supporting  improved  instruction. 

There  are  additional  ways  in  which  KM  programs  and  structured 
training  could  be  mutually  supportive.  For  example,  CALL  products, 
including  videos,  could  augment  DL-collaborative  and  resident  instruc¬ 
tion.  In  addition,  leader  courses  could  teach  students  the  use  of  KM 
learing  delivery  capabilities  while  delivering  training.  For  example,  stu¬ 
dents  could  be  coached  to  use  CALL  and  BCKS  sites  as  a  source  for 
practical  exercise  solutions  and  encouraged  to  become  active  members 
of  the  appropriate  BCKS  forums.  Such  use  would  also  allow  school 
SMEs  to  assess  the  effectiveness  of  these  sites,  and  provide  constructive 
feedback  for  KM  site  and  program  improvement. 

Mutual  support  could  also  help  in  the  development  of  improved 
collaborative  technologies  and  methods  and  of  a  better  understanding 
of  their  potential.  In  this  regard,  effective  forums  and  effective  asyn¬ 
chronous  collaborative  instruction  are  similar  in  many  respects,  e.g., 
threaded  discussions  and  course  practical  exercises.  What  is  learned 
about  one  could  benefit  the  other. 

A  final  important  point  is  the  need  to  coordinate  the  development 
of  integrated  IT  capabilities  to  support  both  KM  and  DL  systems  and 
programs  in  the  same  way,  so  that  the  soldier  in  a  unit  does  not  have  to 
learn  multiple  IT  procedures  for  access  to  each.  Coordination  to  ensure 
that  the  IT  and  bandwidth  capabilities  can  support  both  programs  will 
also  be  important,  especially  as  programs  expand. 


CHAPTER  SEVEN 


Enhancement  of  Key  Management  Functions  to 
Achieve  TADLP  Transformation 


In  a  previous  chapter  we  described  how  the  Army’s  leadership  is  plan¬ 
ning  for  a  greatly  expanded  TADLP  role  that  will  support  the  achieve¬ 
ment  of  its  training  and  readiness  goals.  We  also  outlined  how  an 
enlarged  TADLP  could  potentially  support  this  requirement;  and  con¬ 
cluded  that  a  truly  transformed  TADLP  capability  is  required  to  meet 
the  Army’s  goals. 

Continuing  with  the  final  stage  of  our  analysis  (as  outlined  in 
Figure  1.2),  this  chapter  addresses  our  final  recommendation  for  broad¬ 
ening  TADLP.  Specifically,  it  describes  the  assessment,  planning,  coor¬ 
dination,  and  implementation  actions  that  are  necessary  to  achieve 
TADLP  transformation.  We  focus  on  actions  for  the  TCM  because  the 
TCM  for  TADLP  has  the  TRADOC  staff  responsibility  to  “develop 
and  implement  policies,  plans,  and  programs  for  TADLP  throughout 
the  Army  training  environment.”1  Given  this  responsibility,  the  TCM 
for  TADLP  has  the  central  role  in  DL  management  activities. 

We  specify  four  key  functions  that  need  enhancement  to  effectively 
achieve  TADLP  transformation  and  suggest  directions  that  the  TADLP 
TCM  could  take  to  reach  that  goal  for  the  TRADOC  commander. 

The  key  functions  we  discuss  are: 

•  Evaluate  and  assess  ongoing  TADLP. 

•  Develop  concepts,  plans,  and  directives  for  TADLP  transformation. 


1  TRADOC  Regulation  10-5-4,  Organizations  and  Functions. 
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•  Implement  a  spiral  development  approach  to  achieve  evolutionary 
transformation. 

•  Perform  combat  developer  role  to  ensure  that  needed  support  is 
provided. 

A  fifth  important  function,  integration  of  TADLP  and  knowledge 
management  learning  delivery  to  provide  the  Army  a  more  comprehen¬ 
sive  distance  learning  capability,  is  covered  in  the  previous  chapter. 


Coordination,  Integration,  and  Collaboration  Across  the 
Army  Are  Needed  to  Accomplish  Functions 

The  TCM  will  need  to  coordinate  the  development  and  implementa¬ 
tion  of  a  transformed  TADLP  approach  across  a  wide  community  of 
stakeholders  and  array  of  issues.  Below  we  describe  the  key  stakehold¬ 
ers  and  relevant  issues  to  be  coordinated  and  integrated. 

Headquarters,  Department  of  the  Army.  Under  the  G-3/5/7 
Director  of  Training  (DOT),  the  Institutional  Training  Division  has 
overall  staff  responsibility  for  institutional  training  and  TADLP,  while 
the  Leader  Development  Division  has  primary  DA  staff  responsibility 
for  leader  development  program  guidance  and  policy  support,  as  well 
as  for  ensuring  that  these  programs  support  an  overall  Army  leader 
training  and  development  strategy. 

But  many  other  staff  elements  are  also  involved  and  interested  in 
training  and  leader  development,  especially  the  Army  G-l,  which  has 
primary  responsibility  for  personnel  readiness. 

The  TCM  should  coordinate  with  the  DOT  to  ensure  that  appro¬ 
priate  Army-wide  policies  are  in  place  to  ensure  TADLP  success,  and 
that  the  DOT  has  the  needed  information  and  data  to  formulate  and 
make  the  case  for  program  resources. 

Headquarters,  TRADOC.  The  TRADOC  commander  is  respon¬ 
sible  for  the  execution  of  institutional  training.  The  TCM  should 
coordinate  within  this  headquarters  to  ensure  that  a  feasible  role 
for  TADLP  is  defined,  that  this  role  and  its  priority  are  understood 
throughout  TRADOC,  and  that  TADLP  is  properly  resourced  with 
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staffing  and  the  support  of  subordinate  schools  and  centers  to  accom¬ 
plish  its  mission. 

Army  Training  Support  Center  (ATSC).  ATSC  is  a  TRADOC  field 
operating  activity.  It  supports  TADLP  efforts  by  providing  DL  man¬ 
agement  support,  DL  Education  and  Training  Products  (DLETP)  con¬ 
tact  services,  DL  testing,  and  development  of  DL  technical  standards 
and  specifications.  The  TCM  should  coordinate  with  the  commander 
of  the  ATSC  to  ensure  that  ATSC’s  efforts  effectively  support  TADLP. 

Combined  Arms  Center  (CAC).  The  CAC  commander  has  the 
TRADOC  lead  for  leader  development  and  functional  training — the 
two  types  of  courses  most  heavily  supported  by  TADLP.  The  Army 
Leade  Development  Program  is  the  responsibility  of  TRADOC,  and  a 
program  office  has  recently  been  set  up  under  CAC. 

Also  under  CAC  is  CAC-Knowledge  (CAC-K),  an  organization 
that  oversees  TRADOC’s  knowledge  development  and  knowledge 
management  learning  delivery  efforts.  CAC-K’s  efforts  to  distribute 
unstructured  learning  to  the  Army  complement  TADLP’s  efforts  to 
give  the  Army  a  comprehensive  DL  capability.  As  we  discussed  in  the 
previous  chapter,  the  TCM  should  coordinate  with  CAC-K  to  ensure 
mutual  support  and  overall  Army  benefit. 

Supporting  Army  Component  Commands,  Army  Service  Sup¬ 
port  Commands.  Numerous  Army  organizations  support  the  Army’s 
ability  to  conduct  DL.  For  example,  the  Army  Materiel  Command’s 
(AMC’s)  Distributed  Learning  System  (DLS)  Program  Manager  devel¬ 
ops  the  ALMS  that  supports  DL  and  manages  the  distance  training 
facilities.  The  Army  G-6  and  the  Installation  Management  Com¬ 
mand’s  (IMCOM)  Directorates  of  Information  Management  (DOIM) 
provide  the  network  connectivity  to  allow  web-based  DL  delivery.  The 
TCM  should  communicate  these  needs  and  coordinate  to  ensure  that 
they  are  being  addressed. 

Supported  Units.  The  customers  for  DL  are  the  soldiers  and  lead¬ 
ers  who  take  courses  and  the  commanders  who  benefit  by  having  grad¬ 
uates  able  to  successfully  perform  organization  missions  and  tasks.  For 
the  TCM,  this  requires  continuous  liaison  with  troop-owning  com¬ 
mands,  both  AC  and  RC,  development  of  policies  governing  the  pro- 
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vision  of  TADLP  instruction  to  soldiers,  and  program  adjustment  to 
provide  better  support. 

Schools  and  Other  Training  Organizations.  These  are  the  organi¬ 
zations  that  design,  develop,  and  implement  DL  and  resident  courses 
to  meet  the  needs  of  their  proponent  customer  organizations  as  out¬ 
lined  by  DA  and  TRADOC  regulations,  policies,  and  other  guidance. 
This  includes  not  only  TRADOC,  but  also  RC  institutional  training 
organizations.  For  NCOES,  the  U.S.  Army  Sergeants  Major  Academy 
is  responsible  for  developing  common  core  instruction,  and  the  propo¬ 
nent  schools  are  responsible  for  MOS-specific  instruction. 

This  listing  is  far  from  inclusive.  The  key  point  is  that  the  coor¬ 
dination  and  integration  of  DL  into  an  overall  leader  training  and 
development  construct  involves  many  important  stakeholders  within 
and  outside  of  TRADOC.  Thus  the  TCM’s  coordination  role  is  large, 
important,  and  complex. 


Evaluate  and  Assess  Ongoing  TADLP 

Key  to  program  management  and  improvement  is  the  establishment  of 
a  means  to  evaluate  and  assess  the  program  to  support  needed  improve¬ 
ments  and  timely  adjustments.  This  is  especially  critical  considering 
the  increased  expectations  for  TADLP  to  expand  its  support  of  train¬ 
ing  and  leader  development  strategies. 

A  Structured,  Supported  Evaluation  and  Assessment  Program  Is 
Needed 

Our  overall  finding  is  that  no  systematic  evaluation  and  assessment 
program  is  in  place  at  the  program  level.  Clearly,  program  evaluation 
and  assessment  is  a  critical  TCM  function,  not  only  for  program  man¬ 
agement,  but  for  making  an  effective  case  for  resources  and  support. 
We  argue  that  this  function  is  one  needing  immediate  improvement, 
and  that  an  effective  system  could  be  established  based  on  the  method¬ 
ology  outlined  in  Chapter  Three. 

The  assessment  system  outlined  in  Chapter  Three  is  based  on  five 
measures  of  effectiveness:  program  impact,  efficiency,  quality,  cycle  time, 
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and  responsiveness.  Establishing  such  a  system  depends  on  collecting 
data  to  form  the  measures  of  effectiveness.  In  previous  sections  of  this 
report  we  presented  sources  and  methods  for  such  data-collection  efforts. 
In  this  section  we  present  our  further  suggestions  on  how  TADLP  eval¬ 
uations  and  assessments  and  supporting  data-collection  efforts  can  be 
expanded  and  improved. 

Assessments  Should  Address  Broader  Quality  and  Responsiveness 
Issues 

As  discussed  in  Chapter  Three,  the  evaluation  and  assessment  of  DL 
courseware  quality  are  primarily  decentralized.  Proponent  schools 
are  responsible  for  validating  IMI  courseware  to  determine  whether 
its  content  is  consistent  with  course  learning  objectives.  However,  our 
observations  indicate  that  there  is  a  need  for  a  broader  range  of  these 
evaluation  efforts.  They  should  be  more  comprehensive  and  based  on 
standardized  measures  to  enable  synthesis  of  results  across  courses  or 
schools.  Guidance  on  how  to  conduct  such  evaluations  also  needs  to 
be  updated.2 

At  TADLP  level,  quality  and  responsiveness  assessments  are 
limited.  The  only  evaluations  focus  on  technical  compliance  of  course¬ 
ware  to  SCORM  and  ALMS  standards;  there  are  no  evaluations  to 
provide  information  about  the  quality  or  currency  of  instructional 
material.  At  program  level,  we  found  no  systematic  effort  to  oversee 
or  collect  data  from  the  proponent  IMI  courseware  quality-evaluation 
efforts.  As  a  result,  TADLP  does  not  have  sufficient  data  or  data  inte¬ 
gration  to  gauge  overall  IMI  courseware  quality  or  the  responsiveness 
of  the  courseware  development  system  to  keep  material  current.  More- 


2  The  guidance  on  IMI  courseware  development  is  in  TRADOC  Pamphlet  350-70-2,  Mul¬ 
timedia  Courseware  Development  Guide,  June  2003.  The  guidance  in  terms  of  quality  control 
and  courseware  validation  is  general.  For  example,  the  discussion  of  courseware  validation 
states  that  “Validation  is  an  essential  step  in  the  development  of  all  training  materials,”  but 
provides  no  effective  direction  on  how  to  accomplish  this  essential  process.  The  training 
developer  community  in  TRADOC  is  aware  of  shortfalls  in  its  overarching  training  devel¬ 
opment  regulation,  TRADOC  Regulation  350-70,  Systems  Approach  to  Training  Manage¬ 
ment,  Processes,  and  Products,  March  1999,  as  well  as  supporting  pamphlets,  and  is  in  the 
process  of  revising  these  publications. 
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over,  we  found  no  effort  of  any  kind  to  evaluate  or  assess  the  effective¬ 
ness  of  other  types  of  distance  learning,  such  as  VTT  courses. 

Limited  plans  appear  to  be  in  place  to  enhance  TADLP  level 
quality  assessments.  A  2006  draft  of  the  Campaign  Plan  for  TADLP 
describes  a  “new”  Training  Effectiveness  Assessment  (TEA)  program 
in  which  the  TCM  will  “develop,  test,  and  implement  a  system  for  DL 
TEA.  The  system  will  require  graduates  and  supervisors  to  assess  the 
graduates’  abilities  to  accomplish  tasks  taught  through  DL.”3  Evalua¬ 
tion  will  depend  largely  on  the  use  of  the  AUTOGEN  survey  system. 
AUTOGEN  surveys  ask  course  graduates  and  their  supervisors  to  pro¬ 
vide  judgments  (using  “yes”  and  “no”  response  options)  about  the  extent 
to  which  the  graduates  can  perform  tasks  to  standard.  Proponent  school 
staff  also  can  add  customized  questions  to  the  survey.  AUTOGEN 
surveys  are  typically  fielded  six  months  following  course  completion. 
Reports  show,  for  each  task,  the  percentages  of  graduates  and  supervi¬ 
sors  who  said  that  the  task  was  being  performed  adequately  and  also 
highlight  discrepancies  between  the  two  groups’  responses.  Reports 
also  include  the  number  of  participants  sampled,  response  rates,  some 
demographic  characteristics,  and  reasons  for  nonresponses. 

AUTOGEN  can  serve  as  a  tool  to  identify  areas  for  improvement 
in  training.  Improvement  might  be  indicated  by  a  low  percentage  of 
“yes”  responses,  showing  that  few  supervisors  or  graduates  report  that 
graduates  were  trained  to  standard,  or  by  large  discrepancies  between 
graduates’  and  supervisors’  judgments  as  to  whether  graduates  were 
trained  to  standard. 

In  our  assessment,  however,  AUTOGEN  is  not  fully  suited  to 
meeting  TADLP  needs  for  evaluation.  The  most  critical  issue  is  that, 
for  most  courses,  AUTOGEN  does  not  provide  information  about  the 
quality  of  DL  because:  (1)  AUTOGEN  surveys  are  administered  after 
students  complete  training  and  return  to  their  units,  and  (2)  DL  is 
typically  used  in  a  blended  model.  Therefore,  responses  to  AUTOGEN 
surveys  do  not  distinguish  learning  that  occurred  in  DL  from  learn¬ 
ing  that  occurred  in  resident  training  or  during  unit  training  after  the 


3  U.S.  Army  Training  and  Doctrine  Command,  TRADOC  Campaign  Plan  with  Change  1, 
June  2006. 
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completion  of  resident  training.  Such  a  distinction  would  be  possible 
only  if:  (1)  a  course  was  taught  entirely  through  DL;  or  (2)  tasks  in  a 
blended  course  were  taught  exclusively  through  DL. 

There  are  a  number  of  other  issues  with  AUTOGEN  that  pertain 
to  DL  as  well  as  other  training  methods: 

•  Supervisors  are  asked  about  performance  of  graduates  as  a  whole 
rather  than  performance  of  graduates  on  an  individual  basis. 
Therefore,  AUTOGEN  cannot  link  individuals’  performance  in 
training  with  their  performance  on  the  job,  which  is  an  important 
component  of  evaluating  training  effectiveness. 

•  Participants  are  asked  about  the  tasks  on  which  graduates  are 
trained  to  standard.  Despite  the  existence  of  defined  standards 
for  tasks  in  soldier  training  publications,  there  may  be  different 
ways  that  respondents  interpret  “to  standard.”  This  renders  the 
results  ambiguous  or  noncomparable  between  graduates  and  their 
supervisors  or  across  different  supervisors. 

•  AUTOGEN  collects  only  subjective  ratings;  it  does  not  include 
objective  measures  of  learning  or  job  performance.  Moreover,  in 
some  cases,  supervisors  in  the  units  do  not  have  enough  informa¬ 
tion  to  even  subjectively  evaluate  graduates’  job  performance. 

•  Course  graduates  often  do  not  perform  the  tasks  on  which  they 
were  trained.  Therefore,  even  if  supervisors  are  well  informed 
about  graduates’  job  performance,  the  graduates  may  not  be  per¬ 
forming  the  tasks  taught  in  the  course. 

•  Collecting  ratings  from  graduates  directly  is  problematic  in  that 
self-ratings  can  be  subject  to  social  desirability  biases. 

•  Our  interviews  with  proponent  school  staff  suggest  low  to  mixed 
receptivity  toward  the  AUTOGEN  system.  Some  staff  report  low 
response  rates  for  AUTOGEN  surveys,  particularly  from  person¬ 
nel  who  are  located  overseas  with  limited  Internet  access. 

Currently,  AUTOGEN  surveys  are  developed  and  administered 
by  the  proponent  schools,  without  any  aggregation  of  results  at  the  pro¬ 
gram  level.  As  a  result,  TRADOC  does  not  have  information  such  as 
school  compliance,  graduate  and  supervisor  response  rates,  average  rat- 
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ings,  and  so  forth.  Without  aggregate  data,  there  is  no  way,  at  TADLP 
managerial  levels,  to  know  the  value  of  the  AUTOGEN  system  or  to 
use  the  data  to  evaluate  courseware  quality. 

While  this  review  of  AUTOGEN  highlights  its  limitations, 
we  think  it  could  be  developed  to  play  a  useful  role  in  an  evaluation 
and  assessment  program,  both  at  proponent  schools  and,  if  the  data 
are  shaped  and  shared,  at  TADLP  managerial  levels.  It  does  provide 
potentially  useful  data  on  how  well  tasks  are  trained  in  stand-alone 
courses.  Moreover,  items  can  and  should  be  added  to  give  information 
on  how  students  take  DL  courses  and  how  well  they  are  supported  in 
this  effort.  For  example,  students  could  be  queried  about  whether  they 
took  courses  on  personal  or  duty  time,  whether  they  had  problems 
connecting  to  the  course  sites  or  with  courseware  navigation,  and  how 
courses  could  be  changed  to  better  serve  their  needs  and  constraints. 
They  also  could  be  asked  about  their  views  of  how  far  and  by  what 
means  TADLP  benefits  could  be  advanced. 

Moreover,  the  focus  of  AUTOGEN  is  on  course  quality,  but  the 
TCM  has  a  much  wider  range  of  responsibilities  and  must  meet  the 
needs  of  diverse  customers  or  stakeholders  including  students,  unit 
commanders,  and  school  staff.  This  calls  for  a  much  broader  program 
of  quality  and  responsiveness  evaluation. 

Table  7.1  identifies  an  approach  and  data  that  could  support  a  more 
comprehensive  program  to  evaluate  DL  course  quality  and  responsive¬ 
ness.  For  example,  evaluations  of  the  content  of  courseware  by  SMEs 
and  surveys  of  commanders  could  be  used  to  evaluate  whether  DL  is 
meeting  current  training  needs;  cycle  time  for  courseware  development 
could  provide  information  about  the  currency  of  courseware. 

In  Chapter  Three  we  outlined  our  approach  to  evaluating  the 
quality  of  IMI  courseware  against  three  important  criteria:  techni¬ 
cal,  production  quality,  and  pedagogical.  As  outlined  in  Chapter  Four, 
we  think  an  adapted  version  of  our  IMI  quality  assessment  should  be 
implemented  at  TADLP  level,  but  enhancement  would  further  increase 
its  benefit.  An  enhanced  evaluation  should  involve  taking  the  course  as 
the  students  do  (mainly  online),  and  would  include  evaluation  of  test 
adequacy,  which  we  were  not  able  to  do  for  the  course  modules  we  eval¬ 
uated.  Some  of  the  data  we  collected  on  DL  course  usage  could  pro- 
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Table  7.1 

Approach  and  Data  to  Support  Evaluation  and  Assessment  of  Quality  of 
TADLP  Training 


Criterion 


Metric/Data 


Does  IMI  courseware 
have  appropriate 
instructional  design: 
technical,  production, 
and  pedagogical 
quality? 


Is  TADLP  meeting 
current  training  needs, 
i.e.,  is  it  providing 
training  for  critical 
knowledge  and  skills 
and  is  training  up  to 
date? 

How  well  are  students 
learning? 


•  Adoption  of  RAND's  approach  as  described  in 
Chapters  Three  and  Four 

-  Take  online  and  include  test  review 

•  Enrollment  and  dropout  rates 

•  Student  surveys — potentially  by  an  expanded 
AUTOGEN  system  addressing  the  general  adequacy 
of  DL  courseware  and  methods 

•  SME  evaluations  of  courseware  content 

•  Enrollment  and  dropout  rates 

•  Student  and  commander  surveys  on  additional 
learning  needs  for  stand-alone  DL  courses 

•  Cycle  time  for  courseware  development 


•  Tests  of  learning  and  performance  on  critical  tasks/ 
TLO  following  DL  portions  of  training 

•  Student  and  supervisor  surveys  of  task  performance 
in  DL  stand-alone  courses 

•  Administration  of  tests  measuring  knowledge 
retention  of  concepts  (ELO)  taught  in  DL  prior  to 
start  of  resident  phase  in  courses  using  the  phased 
approach  to  training 

•  Instructor  and  student  surveys 


vide  indicators  of  courseware  quality  issues.  For  example,  high  dropout 
levels  could  indicate  technical  or  content  issues.  Graduates  could  also 
be  surveyed  about  both  their  overall  satisfaction  with  TADLP  courses, 
and  their  satisfaction  in  specific  areas. 

Additional  evaluations  and  data  collection  would  also  be  neces¬ 
sary  to  provide  a  complete  understanding  of  course  quality.  Of  primary 
importance  would  be  checks  of  courseware  completeness,  currency, 
and  accuracy.  SMEs  would  be  a  primary  source  for  such  evaluations. 
While  these  evaluations  are,  and  should  be,  the  responsibility  of  the 
proponent  school,  they  could  be  selectively  done  at  TRADOC  level  as 
well,  for  purposes  of  monitoring  the  overall  program. 

Also  of  great  importance  would  be  an  evaluation  of  the  extent 
to  which  course  learning  objectives  were  attained,  i.e.,  whether  the 
students  actually  learned  what  the  course  was  designed  to  teach.  Here 
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the  primary  mechanisms  would  be  tests  of  student  performance  con¬ 
ducted  soon  after  course  completion.  For  TADLP  courses  in  which 
learning  objectives  are  ELOs  in  preparation  for  face-to-face  instruc¬ 
tion,  this  could  be  done  by  testing  the  knowledge  of  the  subject  matter 
taught  in  the  DL  phase  at  the  beginning  of  resident  training,  and 
instructors  and  students  could  be  surveyed  or  interviewed  during  or 
at  the  end  of  the  face-to-face  portions  about  the  adequacy  of  TADLP 
preparation. 

Assessments  Should  Address  the  Amount  of  Training  Developed 
and  Conducted 

In  Chapter  Three  we  presented  data  on  the  impact  of  TADLP  in  terms 
of  the  amount  of  training  conducted  and  the  proportion  of  overall  insti¬ 
tutional  training  represented  by  TADLP  courses.  We  think  that  this 
evaluation  was  useful  both  in  terms  of  showing  program  impact  objec¬ 
tively  and  showing  where  improvements  might  increase  the  impact. 

Our  evaluation  was  limited  by  the  data  sets  available.  With 
improved  data,  more  would  be  possible.  For  example,  we  could  not 
count  the  number  of  active,  ARNG,  or  Army  Reserve  enrollees  or  grad¬ 
uates  and  there  was  no  way,  given  facility  utilization  records,  to  quan¬ 
tify  the  amount  of  facilities  usage  that  supported  DL  course  execution, 
or  to  determine  whether  these  facilities  were  used  and  needed  by  stu¬ 
dents  to  take  IMI  courses.  Flowever,  we  think  that  setting  up  enhanced 
tracking  and  analysis  methods  could  greatly  increase  TADLP’s  abil¬ 
ity  to  conduct  these  types  of  analysis.  For  example,  ATRRS  could  be 
cross-referenced  with  personnel  databases,  and  digital  usage  logs  could 
be  maintained  at  DTFs. 

Therefore,  we  think  our  methodology  for  evaluating  TADLP 
should  be  incorporated  into  a  larger  TADLP  evaluation  program  and 
expanded  to  further  the  benefit.  In  Table  7.2  we  show  some  additional 
approaches  that  could  be  taken. 

RAND’s  analysis  of  courseware  usage  statistics  can  provide  other 
indicators  of  training  benefits  (or  the  lack  thereof)  and  the  efficiency  of 
TADLP  training.  For  example,  RAND  found  that  many  DL  courses 
were  developed  but  never  used  or  had  short  shelf  lives;  these  are  impor¬ 
tant  indicators  of  how  well  TADLP,  contractors,  and  schools  execute 
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Table  7.2 

Approach  and  Data  to  Support  Evaluation  and  Assessment  of  the  Amount 
of  Training  Delivered  by  TADLP 


Criterion 


Approach  and  Data 


What  DL  training  is  being 
executed? 


Is  the  training  getting  to 
the  soldiers  who  need  it? 

Is  training  available  when 
and  where  needed? 

What  is  the  courseware 
production  amount,  cycle 
time,  and  lifespan? 


•  Number  of  TADLP  enrollees  and  graduates  by: 

-  course 

-  proponent 

-  course  type  (e.g.  VTT,  I M I,  asynchronous 
collaborative) 

-  student  component 

•  Percentage  of  TADLP  course  execution  compared 
to  overall  institutional  course  execution  by: 

-  course  and  student 

-  course  type 

-  proponent 

•  Enrollment  rates  versus  requirements 

•  Backlog  of  unqualified  leaders 

•  Surveys  of  commanders  and  students 

•  TRADOC  contract  records 

•  ATRRS 


their  TADLP  mission  and  an  important  factor  in  a  cost-benefit  approach 
to  program  assessment.  Low  student  enrollments  and/or  high  dropout 
rates  could  be  indicators  of  low  courseware  quality,  and  low  enrollments 
could  also  indicate  that  courses  are  not  available  when  needed.  Ulti¬ 
mately,  follow-up  investigation  would  be  needed  to  systematically  iden¬ 
tify  the  reasons  behind  these  issues.  Surveys  and/or  interviews  could 
be  conducted  of  school  staff  regarding  unused  courses;  of  students  who 
drop  out  of  courses  regarding  their  reasons  for  doing  so;  and  of  poten¬ 
tial  students  regarding  reasons  for  nonenrollment. 

Better  Collection  of  Cost  Data  Could  Enhance  Efficiency 
Assessments 

Efficiency  assessment  requires  the  comparison  of  program  costs  and 
outputs — the  quality  of  courses  and  their  impact.  During  our  evalua¬ 
tion  of  TADLP,  we  found  data  on  the  contractor  costs  for  developing 
IMI  courseware  and  costs  of  supporting  the  operation  of  DTF  facili- 
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ties,  but  only  limited  data  were  available  in  other  important  areas, 
e.g.,  on  the  total  costs  of  IMI  courseware  or  of  VTT-delivered  DL 
courses. 

Although  contractor  IMI  development  costs  per  course  were 
captured,  our  examination  of  courseware  quality  indicated  that  there 
is  a  level  of  uncertainty  about  what  those  costs  really  mean.  Course¬ 
ware  contract  rates  are  determined  by  interactivity  levels  and  hours, 
but  our  sampling,  though  not  systematic,  found  that  much  of  the 
courseware  was  at  lower  interactivity  levels  than  contracted  for,  and 
there  were  indications  that  some  lessons  could  be  completed  in  fewer 
hours  than  indicated  by  the  POI.  Given  this  finding,  we  recommend 
that,  as  a  minimum,  interactivity  levels  and  length  be  integrated  into 
the  content  assessment  of  courseware  recommended  in  Chapter  Four. 
Further,  the  TCM  should  ensure  that  a  check  on  interactivity  levels 
and  length  becomes  a  component  of  proponent  school  IMI  validation 
processes. 

Table  7.3  describes  an  approach  and  the  data  needed  to  support 
an  evaluation  and  assessment  of  the  costs  of  TADLP  training. 

During  our  examination  of  IMI  production  processes  and  inter¬ 
views  with  school  and  IMI  contractors,  we  found  that  the  major  issues 
with  IMI  courseware  production  quality  and  production  time  arose 
from  lack  of  adequate  school  support,  specifically  GFI/GFM,  subject 
matter  experts,  and  front-end  analysis.  Clearly,  a  better  understanding 


Table  7.3 

Approach  and  Data  to  Support  Evaluation  and  Assessment  of  the  Costs  of 
TADLP  Training 


Criterion 

Metric  or  Approach 

What  are  the  costs  of 

•  The  cost  of  IMI  courseware — by  interactivity  level  and 

DL  development  and 

hour 

implementation? 

•  Supporting  costs,  including: 

-  instructor  and  other  staff  time  for  development  and 
execution 

-  student  time  to  take  courses 

-  travel:  student,  instructor,  and  staff 

-  equipment 

-  facilities 
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of  these  requirements  and  shortfalls  is  important  to  future  program 
improvement.  Collecting  staff  manpower  cost  data  in  such  areas  across 
all  proponent  courses  would  require  a  major  effort,  but  a  case  study 
approach,  examining  development  and  execution  of  DL  for  a  selected 
set  of  courses,  would  be  feasible  and  would  provide  for  better  estimates 
of  critical  support  requirements  for  IMI. 

Moreover,  in  light  of  our  recommendation  for  expanding  DL  to 
include  more  than  stand-alone  IMI,  cost  data  on  MTT,  VTT,  and 
asynchronous  DL  approaches  should  be  collected.  Especially  valuable 
could  be  collection  of  data  to  allow  a  comparison  of  DL  versus  resident 
course  costs. 

Establishing  a  Structured,  Supported  Evaluation  and  Assessment 
Program 

We  recommend  that  the  TCM  start  by  immediately  establishing  a 
program-level  evaluation  and  assessment  program  and  staffing  an 
office  directly  responsible  for  this  function.  Based  on  our  efforts,  we 
think  that  RAND’s  approach  could  be  quickly  implemented  and 
expanded  with  a  small,  two-  to  three-person  staff  element. 

This  evaluation  and  assessment  program  should  take  the  business 
management  approach  we  discussed  above.  While  a  strict  cost-benefit 
analysis  approach  is  not  feasible,  given  the  somewhat  subjective  nature 
of  TADLP  benefit  assessments  and  the  difficulty  of  quantifying  sup¬ 
porting  costs,  an  approach  that  reasonably  estimates  costs  and  benefits 
is  possible. 

A  complementary  need  is  to  establish  standardized  counterpart 
proponent  school  programs.  In  the  current,  decentralized  process, 
evaluations  are  generally  localized,  resulting  in  diverse  approaches  that 
cannot  be  aggregated  at  the  program  level.  While  training  evaluations 
and  assessments  should  continue  to  be  conducted  by  the  proponent 
schools,  an  integrated  approach  is  needed.  We  recommend  using  a  col¬ 
laborative  process  to  designing  such  an  approach  by  drawing  on  exper¬ 
tise  from  staff  in  a  subset  of  proponent  schools.  We  suggest  enlisting 
these  staff  to  serve  on  a  temporary  task  force  headed  by  the  TCM 
that  would  be  responsible  for  identifying,  designing,  and  codifying 
standardized  methods  and  metrics  that  can  be  used  across  schools  and 
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courses.4  For  example,  the  Armor  Center’s  methods  of  test  evaluation, 
which  have  shown  benefit  in  identifying  problematic  items,  might  be 
used  to  refine  tests  or,  more  importantly  from  a  strategic  perspective, 
to  identify  areas  where  course  improvement  is  needed.  These  meth¬ 
ods  could  be  standardized  and  extended  across  proponent  schools. 
This  task  force  could  also  be  responsible  for  disseminating  information 
about  these  methods  to  the  community  of  proponent  schools  through 
TRADOC  pamphlets,  handbooks,  or  other  similar  documentation  as 
well  as  workshops  or  similar  training  provided  to  other  school  staff. 

A  collaborative  approach  that  involves  staff  from  the  proponent 
schools  offers  a  number  of  advantages,  including: 

•  Involving  staff  who  know  what  will  work  and  not  work  in  their 
organizations. 

•  Enlarging  the  pool  of  staff  available  to  contribute  to  development 
of  new  efforts. 

•  Enhancing  commitment  to  decisions. 

•  Providing  an  opportunity  for  learning  as  participants  share  their 
experiences. 

•  Providing  credible  “PR”  to  market  new  efforts  to  others  in  the 
training  community. 

A  second  strategic  issue  for  TADLP  is  to  ensure  that  training  staff 
in  the  proponent  schools  have  the  skills  they  need  to  carry  out  training 
assessments.  We  address  this  topic  in  more  detail  later  in  this  chapter. 


Develop  Revised  Concepts,  Plans,  and  Directives 

Once  an  ongoing  DL  evaluation  effort  is  established,  the  starting  point 
for  transformation  is  to  develop  concepts  and  a  plan  for  expanding  DL 
to  support  a  greater  proportion  of  institutional  training  with  the  larger 


4  It  may  be  appropriate  to  design  assessments  that  enable  integration  but  also  allow  for 
customization.  For  example,  a  standard  set  of  questions  on  student  satisfaction  surveys  can 
enable  TADLP  to  aggregate  results  at  the  program  level.  At  the  same  time,  each  school  may 
wish  to  add  questions  that  are  specific  to  its  context. 
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goal  of  improving  the  Army’s  ability  to  provide  needed,  timely  training 
to  support  operational  readiness. 

Figure  7.1  illustrates  the  inputs  that  should  be  considered  in  devel¬ 
oping  concepts  for  expanding  the  use  of  DL  to  better  meet  changing 
requirements.  A  key  element  is  that  concepts  leverage  the  potential  of 
new  DL  technologies  and  methods  discussed  in  Chapter  Five. 

While  the  specific  blends  of  DL  types,  MTT,  and  resident  learn¬ 
ing  will  vary  by  the  type  and  number  of  tasks  and  learning  objectives 
in  each  course,  we  think  a  consensus  can  be  reached  to  develop  general 
guidelines  on  what  tasks,  skills,  and  types  of  knowledge  could  best  be 
taught  by  each  modality.  For  example,  under  such  a  construct,  prac¬ 
tical  exercises  and  tests  of  critical  hands-on  equipment  tasks  would 
most  likely  be  accomplished  by  face-to-face  modes  at  resident  schools 
or  by  MTTs.  Knowledge  of  tactical  concepts  as  an  ELO  could  often 
be  delivered  and  tested  by  asynchronous  IMI,  but  the  understanding 
and  application  of  these  concepts  to  actual  scenarios  would  generally 
require  collaborative  DL,  realistic  games  or  simulations,  or  face-to-face 
modalities  to  be  done  to  standard.  In  general,  any  instruction  that  is 
currently  done  in  a  classroom  could  be  considered  a  target  for  IMI 


Figure  7.1 

Developing  a  Concept  for  a  Transformed  TADLP  Capability 
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and  collaborative  DL  methods,  although  careful  attention  is  warranted 
regarding  the  higher-level  learning  typically  required  for  successful 
achievement  of  TLOs. 

These  concepts  should  be  based  on  reasonable  assumptions  about 
the  Reset  environment  and  unit  readiness  needs.  For  example,  for 
many  courses  it  would  be  reasonable  to  assume  that  unit  equipment 
and  assistant  instructors  could  be  made  available  to  support  MTT 
course  execution.  Use  of  MTT  with  unit  equipment  could  in  fact  be 
a  preferred  option,  as  there  are  obvious  advantages  to  training  on  the 
actual  type  of  equipment  the  unit  will  use  during  the  collective  train¬ 
ing  that  will  follow  the  Reset  phase  in  the  ARFORGEN  cycle. 

Concepts  Should  Be  Developed  in  Collaboration  with  the  Entire 
Training  Community 

These  concepts  should  be  developed  by  a  knowledgeable  working  group 
that  includes  empowered  representation  from  across  the  instructional 
and  user  community.  The  instructional  community  should  include 
instructors  and  SMEs,  as  well  as  training  developers  and  IMI  contrac¬ 
tors.  Experts  for  civilian  academic,  training,  simulations  and  gaming, 
IMI,  and  IT  communities  should  also  be  brought  in  to  support  the 
effort. 

We  also  think  a  collaborative  process  would  facilitate  other  impor¬ 
tant  outcomes,  including  the  exchange  of  best  methods  and  practices 
and  achievement  of  a  common  understanding  of  the  capabilities  of  the 
various  DL  modalities  and  of  what  can  and  cannot  be  done  by  each. 

Participation  by  the  unit  user  community  is  especially  important. 
Key  to  a  viable  concept  is  an  understanding  of  the  type  and  amount  of 
training  needed,  how  much  is  possible  given  time  and  other  resource 
constraints,  and  the  priorities  assigned  to  different  aspects  of  the  train¬ 
ing.  Even  more  important  is  gaining  consensus  with  unit-owning  com¬ 
mands  concerning  the  equipment,  facility,  and  instructor  support  that 
can  be  reasonably  provided  and  the  policies  that  can  be  established  for 
student  availability. 

This  group  would  develop  more  than  concepts  for  using  DL; 
rather,  the  objective  would  be  to  understand  how  to  effectively  blend 
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DL  and  face-to-face  training  modes.5  The  goal  would  be  to  move  as 
much  TDY  instruction  as  feasible  to  an  exported  mode,  and  to  expand, 
to  the  degree  possible,  the  number  of  critical  tasks  taught,  while  still 
maintaining  standards. 

Based  on  the  concept  developed  by  these  efforts,  the  TCM,  in  con¬ 
junction  with  other  directorate-level  members  of  the  TRADOC  insti¬ 
tutional  training  staff  elements,  would  develop  an  overall  TRADOC 
strategy  for  moving  along  a  path  to  making  real,  responsive  progress 
toward  meeting  the  ACP,  TCR  and  recent  Chief  of  Staff  of  the  Army 
(CSA)  guidance. 

The  TCM  would  develop  implementation  plans  for  these  con¬ 
cepts,  with  the  initial  priority  being  given  to  TDY  courses  needed 
during  Reset  and  to  courses  that  help  meet  the  CSA’s  guidance  on 
NCOES  backlog  reduction.  Planning  would  be  coordinated  within 
TRADOC  and  with  DA,  unit-owning  commands,  and  supporting 
commands.  TRADOC  plans,  agreements  with  unit-owning  and  sup¬ 
porting  commands  and  activities,  and  DA  policies  would  follow. 

Command  Emphasis  and  Direction  Will  Be  Critical 

These  actions  would  culminate  in  TRADOC  commander  directives  to 
proponent  training  organizations  to  begin  implementation,  including 
timelines  and  specific  objectives. 

Experience  shows  that  strong  command  emphasis  is  key.  Although 
there  has  been  broad  guidance  to  move  to  greater  DL  support  for  insti¬ 
tutional  training  since  the  initiation  of  the  DL  program  over  10  years 
ago,  the  level  of  change  implemented  has  been  limited. 

Many  factors  have  contributed  to  this  limited  degree  of  change, 
with  low  funding  levels  for  TADLP  certainly  being  one  of  the  most  sig¬ 
nificant,  but  our  interviews  with  school  staffs  indicate  that  proponent 
schools  have  been  reluctant  to  shift  from  the  current,  almost  exclu¬ 
sively  resident-based  approach  for  active  component  instruction.  There 
is  some  basis  for  this  reluctance,  since  there  is  certainly  inherent  value 
in  traditional  resident  small-group  instructor  methods.  This  includes 


5  TRADOC  would  need  to  provide  additional  guidance  on  how  to  achieve  the  appropriate 
blend  to  achieve  optimized  learning  and  readiness  benefits. 
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the  advantages  of  getting  students  away  from  the  day-to-day  interrup¬ 
tions  of  home-station  activities  and  of  gathering  them  in  groups  whose 
members  have  a  range  of  experiences  to  share  and  who  can  help  each 
other  develop  solutions.  The  notion  of  “If  it  ain’t  broke,  don’t  fix  it” 
has  some  merit,  and  the  traditional  courses  certainly  were  not  broken. 
Rather,  the  issue  is  that  the  training  environment  has  changed  in  many 
ways,  and  training  delivery  must  change  to  ensure  that  needed  training 
is  getting  to  soldiers  in  a  timely  manner. 

Given  the  limited  distance  learning  capabilities  currently  in  place, 
we  conclude  that  much  stronger  command  direction  will  be  needed. 
Our  discussions  of  both  the  Navy’s  IMI  program  (in  Chapter  Four) 
and  the  Special  Forces  DL  initiative  with  regard  to  its  ANCOC  (now 
SLC)  course  (in  Chapter  Five)  indicate  that  clear,  unambiguous  com¬ 
mand  direction  and  support  is  a  prerequisite  for  achieving  real  change. 

Develop  Plans  and  Directives  to  Provide  for  Effective  Transition 

The  transformation  discussed  above  needs  to  be  made  in  a  way  that 
preserves  the  benefits  of  current  training  methods,  and  to  be  carried  out 
in  an  evolutionary  manner  to  avoid  disrupting  an  effective  functioning 
training  system.  This  is  why  a  consensus-building  approach  is  needed, 
not  only  to  develop  a  reasonable  concept  for  institutional  training  trans¬ 
formation,  but  also  to  ensure  that  the  command  guidance  for  executing 
the  transformation  is  feasible  and  allows  for  reasonable  transition. 

In  this  regard,  an  example  is  that  movement  to  a  fully  exported 
NCOES  mode  for  all  soldiers  in  the  unit  ARFORGEN  cycles  is  prob¬ 
ably  not  feasible  or  necessary.  Some  technology-focused  low-density 
MOS-exported  execution  may  not  be  possible  if  needed  equipment, 
simulations,  and  simulators  are  not  available  at  most  home  stations. 
However,  a  reasonable  goal  might  be  to  reduce  TDY  portions  to  a  max¬ 
imum  of  two  or  three  weeks,  with  proponents  having  the  capability  to 
request  exceptions  with  justification.  This  would  largely  achieve  the 
CSA  intent  of  supporting  unit  readiness,  reducing  NCOES  backlog, 
and  limiting  soldier  time  away  from  home  station  during  Reset.  Such 
a  goal  may  be  feasible  for  many  MOSs  if  the  Army  adopts  the  options 
described  in  Chapter  Five,  especially  since  many  hours  seem  to  involve 
“classroom/discussion”  instruction,  and  often  the  equipment  required 
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is  available  from  the  unit.  To  the  degree  that  this  is  the  case,  movement 
to  a  DL  base  is  possible  if  the  following  conditions  are  met: 

•  Most  classroom  training  is  moved  to  IMI  and  collaborative  DL. 

•  Equipment  needed  for  hands-on  training  (MTT  or  otherwise)  is 
available  at  home  stations. 

•  Troop-owning  commands  agree  to  provide  equipment  and  facil¬ 
ity  support  and  establish  policies  that  commanders  make  soldiers 
available  for  NCOES  attendance  during  Reset. 

It  is  possible  to  start  moving  fairly  quickly  to  a  construct  in  which 
NCOES  has  a  far  greater  exportable  component.  This  has  particular 
promise  for  SLC-level  courses,  which  tend  to  have  smaller  hands-on 
components  and  for  which  achieving  minimum  small-group  sizes 
tends  to  make  MTT  approaches  less  feasible.  However,  decisions  about 
how  and  how  much  to  move  toward  exported  training  need  to  be  made 
in  a  collaborative  fashion  and  validated  during  a  spiral  development 
process  (described  later  in  this  chapter). 

Overall,  the  evidence  we  have  gathered  strongly  suggests  that 
responsive  movement  to  a  decreased  residential  component  is  possible. 
This  contention  is  supported  by  the  efforts  in  ALC  Common  Core, 
SF  SLC,  and  TRADOC’s  recent  efforts  to  deliver  many  high-density 
ALCs  by  MTT.  While  there  are  many  advantages  to  resident  courses, 
we  think  that  the  advantage  of  providing  adequate  training  to  more 
soldiers  who  need  it — reducing  the  NCOES  backlog — is  of  greater 
overall  importance. 

We  also  note  that,  even  with  the  movement  of  NCOES  to  an 
exportable  mode,  resident  options  for  MTT  phases  for  some  MOSs 
may  need  to  be  maintained  for  soldiers  not  in  units.  Moreover,  main¬ 
taining  resident  options  would  provide  scheduling  flexibility. 


Implement  a  Spiral  Development  Approach 

The  approach  we  suggest  to  achieving  a  transformed  TADLP  capabil¬ 
ity  is  similar  to  the  spiral  development  process  used  by  the  Army  to 
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develop  new  Table  of  Organization  and  Equipment  (TOE)  organiza¬ 
tions.  The  process  is  one  of  “build  a  little,  test  a  little,  build  a  little.” 
This  process  is  illustrated  in  Figure  7.2. 

Spiral  development  is  a  method  of  rapidly  implementing  change. 
Under  this  process,  an  initial  version  of  a  product  or  program  is  devel¬ 
oped  as  a  kind  of  work  in  progress.  In  this  case,  the  product  might  be 
any  of  a  number  of  new  DL  courses  that  use  an  approach  substantially 
different  from  the  majority  of  existing  DL  courses  and  that  achieve  a 
substantially  increased  usage  of  DL  modalities  in  the  instruction.  The 
assumption  is  that  a  working  prototype  will  be  fielded  early,  allowing 
it  to  be  refined  and  revised  in  stages.  As  shown  in  Figure  7.2,  the  pro¬ 
totype  is  developed  based  on  the  new  concepts  developed  to  address 
needed  improvements  and  new  requirements  and  to  leverage  the  poten¬ 
tial  of  emerging  technologies  and  methods. 

The  prototype  then  is  piloted  using  a  real-world  trial  effort. 
During  the  spiral  development  process,  developers  and  users  observe 
and  assess  the  performance  of  pilot  efforts  in  order  to  understand  what 
works  and  what  does  not.  When  improvement  is  needed,  analysis  is 
performed  to  develop  and  field  a  revised,  improved  version.  Likewise, 
any  helpful  technology  advances  that  have  occurred  can  also  be  incor¬ 
porated  into  the  next  version.  Once  an  improved  version  of  the  prod- 


Figure  7.2 

The  Spiral  Development  Process 
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uct  or  program  is  fielded,  the  process  of  assessment  and  revision  starts 
again.  Thus,  the  process  is  one  of  planned  evolution.  Each  step  results 
in  further  improvements  to  the  product  or  program.  Such  a  process  is 
consistent  with  the  DoD  Training  Transformation  Plan.6 

An  Evolutionary  Approach  Is  Prudent 

A  spiral  development  process  offers  the  most  feasible  approach  to 
enhancing  TADLP’s  contributions  while  allowing  for  adequate  testing 
and  understanding  of  the  capabilities  of  the  various  DL  modes.  Such  a 
process  is  needed  since  the  expanded  DL  program  we  outlined  earlier 
represents  a  significant  change  in  how  training  will  be  developed  and 
conducted,  and  the  effectiveness  of  these  options  has  yet  to  be  dem¬ 
onstrated  or  refined.  For  TRADOC  to  implement  a  fully  expanded 
exportable  training  capability,  the  training  support  community  will 
need  to  work  out  many  details  concerning  training  effectiveness,  orga¬ 
nizational,  and  budget  issues.  Spiral  development  will  allow  such  issues 
to  be  resolved  over  time  while  allowing  change  to  begin  immediately. 
The  idea  of  evolutionary  change  assumes  that  a  product  or  program 
will  change  and  adjust  as  the  Army  conducts  evaluations  and  gains 
experience  and  understanding.  Such  an  approach  also  allows  for  the 
early  involvement  of  stakeholders  in  the  change  process. 

Evolutionary  change  is  best  made  step-wise,  with  early  efforts 
designed  to  gain  as  many  insights  as  possible  into  lessons  that  can 
guide  future  implementation.  Transition  plans  and  even  end-state  goals 
will  likely  need  continuous  adjustment.  A  wide  range  of  participation 
in  early  implementation  efforts  not  only  leads  to  a  wider  range  of  input 
but  also  serves  the  objective  of  consensus  building. 

Evaluation,  Assessment,  and  Adjustment  Are  Key 

For  the  spiral  development  process  to  work,  there  must  be  a  strong 
evaluation  and  assessment  program.  The  evaluation  and  assessment 
processes  presented  earlier  would  all  apply  but  would  need  expansion 
to  support  spiral  development.  Emphasis  is  placed  on  determining 
whether  the  revisions  developed  actually  work  to  improve  course  ben- 


6 


Department  of  Defense,  Strategic  Plan  for  Transforming  DOD  Training,  February  2009. 
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efits  without  unduly  increasing  costs.  In  the  case  of  moving  toward  a 
greater  level  of  exported  delivery  of  NCOES,  it  is  particularly  impor¬ 
tant  to  compare  DL  with  residential  instruction  in  terms  of  the  number 
of  tasks  that  can  be  included  in  the  instruction,  the  adequacy  of  stu¬ 
dent  learning,  and  the  overall  costs. 

Spiral  development  would  also  require  detailed  planning  and  an 
adequate  organizational  structure  to  support  enhanced  evaluation  and 
assessment  activities  such  as  data  collection,  measurement  of  student 
learning,  establishment  of  an  AAR  system,  and  structured  interviews 
with  students,  instructors,  and  training  developers. 

A  model  that  could  be  used  is  the  Stryker  Brigade  implementa¬ 
tion  effort,  which  was  designed  as  a  spiral  development  process.7  CALL 
and  proponent  schools  also  supported  the  evaluation,  assessment,  and 
improvement  process. 

Evaluation  and  assessment  efforts  should  begin  early  enough  to 
capture  all  implementation  costs  and  required  efforts.  Lor  example,  the 
man-days  used  by  instructors  to  develop  and  revise  the  IMI  courseware 
should  be  recorded.  When  coupled  with  evaluation  of  the  effective¬ 
ness  of  this  IMI,  these  data  could  help  estimate  what  future  efforts 
of  this  type  would  require.  These  data  could  also  help  lead  to  a  better 
understanding  of  what  types  of  IMI  would  be  reasonable  for  schools  to 
develop  internally,  and  what  types  would  be  best  to  contract  out.  Other 
important  data  include  the  manpower  effort  needed  to  coordinate  the 
execution  of  exported  training,  both  by  the  school  and  by  home  station 
leaders  and  staffs. 

A  special  area  of  interest  would  be  the  effectiveness  and  instructor 
costs  of  collaborative  DL  methods  in  delivering  classroom  instruction. 
Understanding  what  can  be  done  by  IMI,  asynchronous  collaborative 


7  Specifically,  the  Army  used  a  spiral  development  approach  for  the  Stryker  Brigade  Combat 
Team  (SBCT)  program.  Program  assessment  and  revision  for  the  SBCT  was  the  responsibil¬ 
ity  of  a  dedicated  organization,  the  Brigade  Coordination  Cell  (BCC),  with  approximately 
70  assigned  personnel  headed  by  a  brigadier  general.  The  purpose  of  the  BCC  was  to  coordi¬ 
nate  SBCT  transformation,  assist  in  development  of  transformation  documents,  provide  les¬ 
sons  learned  related  to  current,  interim,  and  objective  forces,  and  help  educate  the  Army  on 
SBCT  and  related  transformation  concepts.  Using  the  spiral  development  process,  the  Army 
was  able  to  implement  rapid,  effective  improvement,  resulting  in  the  development  of  the 
entire  SBCT  system  to  deployable  readiness  levels  less  than  two  years  after  initial  manning. 
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DL,  synchronous  collaborative  DL,  and  actual  face-to-face  instruc¬ 
tion,  as  well  as  how  to  effectively  integrate  these  methods,  is  central 
to  determining  how  far  and  fast  it  will  be  possible  to  move  to  a  greatly 
expanded  use  of  DL. 

Because  resident  options  will  be  maintained,  at  least  initially, 
some  level  of  subjective  comparison  of  effectiveness  would  be  possi¬ 
ble  and  beneficial.  While  rigorous  test-like  comparisons  are  desirable, 
these  are  not  likely  to  be  feasible,  but  SME  judgment  could  still  provide 
a  reasonable  basis  for  making  assessments  and  adjustments. 

Training  Development  Staff  Development  Programs  Should  Be 
Enhanced 

In  order  to  keep  pace  with  the  transformation,  developers  and  instruc¬ 
tors  in  the  proponent  schools  must  have  the  knowledge  and  skills  to 
design,  implement,  and  deliver  many  different  modalities  of  DL.  Cur¬ 
rently,  responsibility  for  the  training  and  development  of  DL  train¬ 
ing  staff  (e.g.,  developers)  is  localized  in  the  proponent  schools,  only 
a  limited  amount  of  structured  TRADOC-level  developer  training  is 
offered,  and  TRADOC/ATSC  does  not  maintain  information  about 
what  educational  opportunities  are  available.  ATSC  offers  some  basic 
courses  in  SAT  and  Automated  Systems  Approach  to  Training  (ASAT) 
for  training  developers  and  offers  several  courses  for  instructors  (Small- 
Group  Instruction,  Army  Basic  Instructor  Course,  Video  Teletrain¬ 
ing,  and  a  newly  developed  course,  Asynchronous  DL  Instruction).8 
We  did  not  find  any  existing  courses — or  plans  to  offer  courses — on 
topics  such  as  media  selection,  models  of  learning  (e.g.,  blended  learn¬ 
ing),  IMI  development,  IMI  delivery,  the  DL  contracting  process,  or 
training  evaluation.  ATSC  does  not  have  the  resources  to  offer  such 
training,  either  by  teaching  it  in  house  or  contracting  it  out.  We  see  a 
need  to  provide  such  training  for  proponent  school  staff  and  anticipate 
that  this  need  will  grow  as  training  requirements  increase,  technolo- 


8  Most  of  these  courses  run  3  or  5  days,  or  24  to  40  hours.  The  Army  Basic  Instructor 
Course  is  longer,  at  80  hours.  In  addition,  the  Advanced  Distributed  Learning  (ADL) 
Co-Lab  offers  several  SCORM  courses. 


160  Making  Improvements  to  The  Army  Distributed  Learning  Program 


gies  change,  and  schools  become  more  directly  involved  in  courseware 
development  rather  than  outsourcing  it  to  contractors. 

We  also  identified  a  variety  of  other  resources  for  training  devel¬ 
opment  staff,  but  these  resources  are  not  available  in  one  central  loca¬ 
tion  and,  while  potentially  helpful,  appear  at  present  to  offer  only  a 
limited  capability: 

•  The  DL  knowledge  network  (DLKN)  on  AKO  has  “knowledge 
centers”  for  AKO  users  who  are  involved  in  DL  courseware  design 
and  development.  According  to  TADLP’s  website  at  the  time  of 
this  research,  the  purpose  of  DLKN  is  to  “share  ideas,  lessons 
learned,  and  developing  guidance  and  specifications;  seek  and 
offer  solutions  to  technical  issues;  and  collaborate  with  others  in 
the  DL  community.”9  DLKN  has  the  potential  to  provide  a  cen¬ 
tralized,  online  location  for  information  exchange,  but  there  are 
only  a  small  number  of  people  who  have  posted  to  the  centers. 
Most  of  the  knowledge  centers  contain  documents  for  use  by  the 
community  at  large,  such  as  copies  of  briefings  from  the  annual 
DL  workshop  from  2006  or  the  DLETP  kickoff  meeting  from 
2006  and  general  DL  technical  alerts.  However,  we  did  not  see 
any  information-seeking  that  one  might  typically  find  in  a  com- 
munity-of-practice  forum. 

•  The  Training  Development  and  Delivery  Directorate  (TDADD) 
webpage  has  links  to  a  variety  of  resources  such  as  job  aids  rel¬ 
evant  to  training  in  general  and  to  IMI  in  particular,  guidance 
for  DL  development,  and  a  draft  “DL  XXI”  guide.10  TDADD 
has  been  reorganized  into  the  Distributed  Learning  Directorate; 
however,  these  resources  for  training  staff  can  be  accessed  only 
through  the  TDADD  webpage. 

Moreover,  as  discussed  previously  in  this  report,  TRADOC  guidance 
and  guidelines  in  the  form  of  regulations,  pamphlets,  handbooks,  and 
other  publications  are  limited  and  dated.  Revisions  are  ongoing,  but 


9  As  of  February  1,  2011:  http://www.tradoc.army.mil/tadlp/links.htm 

10  As  of  February  1,  2011:  http://www.tradoc.army.mi1/g357//tdadd/index.htm 


Enhancement  of  Key  Management  Functions  to  Achieve  TADLP  Transformation  161 


the  effort  is  going  slowly;  given  the  current  staff  and  other  resources 
devoted  to  this  effort,  major  near-term  improvement  is  not  likely. 

Considering  the  dynamic  nature  of  DL  capabilities  and  methods 
and  the  need  to  expand  and  enhance  training  development  capabili¬ 
ties,  we  think  that  a  major  goal  of  the  pilot  assessment  efforts  should 
be  a  structured  effort  to  identify,  observe,  analyze,  and  assess  train¬ 
ing  development  practices.  Such  an  effort  should  codify,  document, 
and  distribute  lessons  learned  and  best  practices,  using  the  AAR  and 
data-collection  efforts  that  will  be  part  of  the  larger  spiral  development 
assessment  effort. 


Perform  Combat  Developer  Role 

An  important  specified  role  for  TCM  is  to  serve  as  the  combat  devel¬ 
oper  for  the  technology  that  supports  TADLP  execution.  For  example, 
execution  of  DL  is  heavily  dependent  on 

•  Learning  management  system  capabilities  to  provide  access, 
tracking,  testing/assessment,  and  record-keeping. 

•  Information  technology  to  provide  hosting  and  serving  of  content, 
required  bandwidth  to  deliver  the  content,  and  interaction  with 
the  content  via  PC  and  browser  “client”  for  the  learner,  both  for 
IMI  content  and  collaborative  tools  for  instructors  and  learners. 

•  IMI  technology  support  as  the  means  to  deliver  instruction  for 
noncollaborative  or  instructor-supported  or  -led  DL,  and  to  main¬ 
tain  that  content  over  the  course  lifecycle. 

TADLP’s  use  of  these  technologies  must  also  continue  to  evolve 
to  take  advantage  of  the  rapid  pace  of  technology  advancement  on  the 
commercial  side  in  such  areas  as  quality  and  delivery  of  IMI,  “serious” 
gaming,  and  wider-bandwidth  uses  of  collaborative  technologies  (such 
as  web-based,  desktop  video  teleconference). 

The  envisioned  expansion  of  the  DL  program  to  a  far  greater 
number  of  learners  will  require  a  greater  capacity  for  students  to  con¬ 
nect  to  IMI  content  and  collaborative  DL  tools.  Increased  student 
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throughput  could  create  demands  for  such  resources  as  bandwidth 
and  workstations  that  exceed  current  and  projected  capabilities.  Thus, 
increased  oversight  and  guidance  by  the  TCM  may  well  be  needed  in 
the  coming  years  to  obtain  increased  resources  and  achieve  the  usage 
goals  for  DL. 

ALMS  Use  Case  Shows  Importance  of  TCM 

Below  we  further  discuss  the  “use  case”  of  the  TCM’s  role  as  combat 
developer  regarding  the  ALMS  in  order  to  explore  how  the  TCM  might 
better  meet  the  needs  of  its  program  and  its  customers.  As  described 
in  Chapter  Two,  the  combat  developer  is  responsible  for  establishing 
requirements  and  representing  the  user  community  through  the  mate¬ 
riel  development  process  of  the  ALMS,  while  the  materiel  developer 
creates  the  solution  based  on  the  requirements.  The  combat  developer 
and  materiel  developer  must  work  together  to  ensure  that  the  ALMS 
requirements  are  appropriately  written,  that  the  solution  developed 
meets  those  requirements,  and  that  the  end  customer’s  needs  are  ulti¬ 
mately  met  by  the  solution. 

To  date,  the  ALMS  program  has  made  significant  investments 
in  achieving  LMS  capabilities  by  using  a  blend  of  COTS  products. 
While  under  development  since  199611  and  fielded  in  2004,  the  ALMS 
currently  is  used  by  few  schools,  including  some  large  schools  that  are 
very  active  in  DL  and  that  develop  high-priority  courses.  Some  large 
schools,  including  ones  that  are  very  active  in  DL,  have  developed  their 
own  LMS  solutions.  Many  schools  have  reported  that  the  originally 
provided  Saba  Enterprise  services  are  not  user-friendly,  do  not  provide 
all  the  capability  they  need,  and  provide  capability  they  do  not  need 
(while  making  the  tool  more  difficult  to  use).  Some  schools  report  that 
they  are  satisfied  with  Blackboard-based  solutions  alone.12  And  others 
also  report  that  they  have  not  been  adequately  funded  to  carry  out  the 


11  The  Army  Distance  Learning  Program  was  approved  in  1996  and  had  an  Operational 
Requirements  Document  (ORD)  published  in  1999,  with  revisions  in  2002.  The  ALMS 
“Engineering  and  Manufacturing  Development”  (EMD)  began  in  August  2000. 

12  Blackboard  Inc.  is  a  software  company  based  in  Washington,  D.C.  that  develops  and 
licenses  software  applications  and  related  services  to  educational  institutions. 
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transition  of  content  from  local  LMSs  to  the  ALMS.  (See  Appendix 
B  for  schools’  comments  on  the  ALMS  that  derived  from  our  survey.) 

Defining  Requirements  That  Meet  User  Requirements  Is  Key 

These  reports  provide  potential  lessons  learned  for  future  TCM  partici¬ 
pation  in  the  materiel  system  requirements  definition  processes.  In  any 
development  effort,  a  key  to  success  is  providing  a  sound  foundation 
of  well-specified  user  needs.  Realistic  requirements  must  be  gathered 
and  synthesized  across  user  stakeholder  groups  (e.g.,  proponent  schools 
and  students)  with  potentially  very  different  needs.  These  requirements 
must  also  be  achievable  within  realistic  system /pro  duct  costs. 

A  key  responsibility  of  the  TCM  as  combat  developer  is  to  pro¬ 
duce  appropriately  scoped  and  defined  requirements  documents  that 
clearly  outline  expected,  realistic,  and  needed  capabilities.  In  the  case 
of  ALMS  this  role  may  not  have  been  fulfilled.  For  example,  some 
features  of  the  ALMS  software  requirements  specification  may  have 
received  more  emphasis  than  warranted.  In  particular,  the  ALMS 
specifications  supplied  by  the  combat  developer  seemed  based  on  an 
assumption  that  all  schools  were  following  the  complex  training  devel¬ 
opment  processes  and  developing  the  products  outlined  in  TRADOC 
Regulation  350-70,  “Systems  Approach  to  Training  Management, 
Processes,  and  Products.”  In  fact,  most  schools  lacked  and  still  lack 
the  required  training  development  and  management  staffs  to  follow 
SAT  processes:  actual  training  development  and  learning  management 
practices  were  far  simpler,  but  they  were  not  reflected  in  the  require¬ 
ments  nor  later  in  the  ALMS  that  was  developed.13 

Testing  to  Ensure  User  Requirements  Are  Met  Is  Also  Key 

Another  key  combat  developer  responsibility,  as  representative  of  the 
user  community,  is  to  conduct  sufficient  testing  to  ensure  that  cus¬ 
tomer’s  needs  are  met  by  the  system  before  and  after  it  is  fielded.  There 


13  TRADOC  Inspector  General  Report,  Soldier  Training  Review,  dated  December  2005.  A 
major  conclusion  was  that  that  while  the  SAT  process  was  basically  sound,  the  staff  resources 
and  training  for  its  implementation  were  lacking.  TRADOC  is  now  in  the  process  of  rewrit¬ 
ing  this  regulation. 
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is  evidence  to  suggest  that  the  TCM  could  have  played  a  stronger  role 
in  assessing  the  ALMS’s  ability  to  meet  school  needs  when  first  imple¬ 
mented.  There  were  several  issues  that  arose  as  the  ALMS  was  fielded 
and  courses  were  developed  to  meet  the  ALMS  data  requirements 
and  requirements  to  conform  with  SCORM.  One  such  issue  was  the 
reported  complexity  of  the  process  by  which  schools  would  prepare, 
test,  and  upload  courses  to  the  ALMS.  Many  schools  also  felt  that 
the  ALMS  requirements  included  SCORM  data  conformance  require¬ 
ments  that  provided  no  local  benefit  to  the  schools,  but  did  impose 
extra  development  and  testing  burdens.  These  requirements  served  to 
slow  the  fielding  of  completed  IMI  courses,  as  content  had  to  be  revised 
and  retested  for  conformance.14  This  example  shows  the  importance  of 
an  effective  acceptance  testing  program  to  identify  and  make  appropri¬ 
ate  modifications  prior  to  fielding.  The  TCM  might  also  have  played  a 
stronger  role  in  this  process  by  specifying  criteria  for  acceptance  testing 
in  a  more  detailed  requirements  document.15 

The  TCM  Can  Also  Play  an  Important  Role  in  Guiding  and  Obtaining 
Resources  for  Product  Improvement 

Looking  ahead,  the  role  of  the  TCM  in  supporting  the  user  commu¬ 
nity  will  become  even  more  important  if  TADLP  is  to  significantly 
expand.  For  example,  fully  supporting  schools  in  migrating  to,  or 
initially  placing  content  on,  the  ALMS  may  well  require  some  addi¬ 
tional  resources  that  were  not  specified  in  the  ALMS  definitions  in 
DLS  Increment  3  Systems  Requirements  Specifications.16  One  way  to 
accomplish  this  is  to  further  evolve  the  concept  of  ALMS  support  to 
schools.  For  example,  the  ALMS  materiel  developer  (PM  DLS)  has 
already  initiated  a  customer  service  approach  that  has  moved  away 


14  An  additional  aspect  of  reported  problems  with  SCORM  conformance  was  that  the 
requirements  were  being  changed  at  many  points  in  the  ALMS  fielding  process. 

15  An  example  would  have  been  specifying  options  for  opting  out  of  certain  aspects  of 
SCORM  conformance  for  content  that  has  a  low  probability  of  being  reused  by  other  train¬ 
ing  developers  in  Army  schools. 

16  Distributed  Learning  System,  System  Requirements  Specifications:  Increment  3,  V2.1-8, 
submitted  by  IBM,  September  8,  2005. 
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from  “training  school  staff  members  to  use  the  ALMS”  to  providing 
much  more  direct  support.  Instead  of  relying  on  school  staff  to  learn 
and  retain  technical  software  expertise  that  is  rarely  used,  the  evolv¬ 
ing  approach  is  to  provide  the  service  for  the  school.  Courses  are  being 
converted  and  uploaded  by  ALMS  customer  support  staff,  who  are  also 
customizing  access  to  the  content  to  meet  the  schools’  requirements. 
These  additional  services  are  resulting  in  customer  satisfaction  among 
new  ALMS  users — primarily  users  just  starting  out  with  their  DL  pro¬ 
gram.  By  supporting  an  expansion  of  this  approach,  the  TCM  might 
also  help  meet  the  needs  of  schools  that  have  evolved  their  own  systems 
to  meet  early  needs  and  who  are  now  reluctant  to  change. 

In  addition,  in  order  to  bring  about  TRADOC-wide  use  of  ALMS 
resources,  the  TCM  may  have  to  provide  additional  support  and  incen¬ 
tives  to  prove  ALMS  capabilities,  to  build  trust  in  the  centralized  solu¬ 
tions,  and  to  move  content  to  the  centralized  solution.  There  are  pos¬ 
sible  blends  of  solutions  that  could  support  incremental  migration;  for 
example,  some  proponents  might  take  advantage  of  central  hosting  of 
IMI  but  local  hosting  of  Blackboard.  A  different  approach  would  be  to 
allow  local  LMS  operation  with  the  requirement  to  provide  automated 
feeds  of  all  ALMS -required  data  to  the  central  ALMS  system.  Such 
data  feeds  would  have  to  meet  the  ALMS  requirements  and  be  funded 
by  the  local  schools  for  development  and  maintenance. 

Another  approach  would  be  for  the  TCM  to  move  toward 
including  “performance-based”  metrics  into  software  requirements, 
e.g.,  measuring  user  satisfaction  or  levels  of  migration  of  content,  over 
time.17  Under  such  an  arrangement,  the  materiel  developer  would  not 
be  responsible  for  simply  providing  the  system  and  training  for  its  use, 
but  would  be  evaluated  based  on  aspects  of  system  usage  and  customer 
satisfaction.18 


17  It  is  important  to  note  that  there  are  well-defined  change-management  processes  to 
guide  the  evolution  of  features/capabilities  of  the  ALMS.  These  are  described  at 
http://www.dls.army.mil/CR_Process.html. 

18  RAND  has  published  research  supporting  the  use  of  “performance-based”  contracts  in 
sustainment  of  Army  materiel  systems. 
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Summary 

Overall,  the  TCM  should  investigate  how  to  improve  its  ability  to  pro¬ 
vide  effective,  continued  general  oversight  of  system  effectiveness  after 
fielding  in  conjunction  with  the  materiel  developer  and  the  user  com¬ 
munity.  There  may  also  be  other  management  or  coordination  efforts 
that  the  TCM  could  champion  to  (a)  increase  the  satisfaction  of  users 
with  materiel  capabilities,  (b)  ensure  that  materiel  developers  are  pro¬ 
ducing  systems  and  products  that  meet  user  needs,  and  (c)  determine 
how  to  obtain  the  increased  resources  needed  to  achieve  DL  usage 
goals  in  the  coming  years. 


Chapter  Conclusions 

In  this  chapter  we  outlined  what  we  think  are  the  important  TADLP 
functions  that  need  enhancement.19  These  functions  are  necessary  for 
institutional  training  transformation  to  achieve  the  improved  train¬ 
ing  support  called  for  by  the  Army’s  leadership  in  the  Army  Cam¬ 
paign  Plan  and  other  guidance.  All  aspects  of  TRADOC  functions 
supporting  Professional  Military  Education,  functional  training  and 
self-development  programs  should  be  reexamined  and  their  priori¬ 
ties  reassessed  in  terms  of  current  critical  needs.  To  the  degree  that 
DL  capabilities  must  be  expanded,  priorities  should  be  revised  and 
resources  reallocated. 

Moreover,  we  point  out  that  a  necessary  change  for  an  enhanced 
DL  capability  is  increased  command  support.  The  TCM  can  and 
should  coordinate  an  effective  transformational  shift  to  an  enhanced 
DL  capability,  but  in  the  end,  success  will  be  a  function  of  the  extent 
to  which  the  Army’s  leadership  emphasizes  the  need  for  change  and 
provides  resources  and  other  support  to  achieve  it. 


19  We  have  specified  functions  without  attempting  to  analyze  the  current  staffing  levels  of 
the  TCM,  TRADOC  Headquarters,  DLS,  or  ATSC.  Reallocation  of  staff  to  implement 
these  functions,  along  with  contract  funding  and  school  support  for  spiral  development, 
would  help  to  support  the  suggested  enhancements. 


CHAPTER  EIGHT 


Conclusions,  Recommendations,  and  Implications 


In  this  report  we  have  examined  the  benefits  of  TADLP  in  terms  of 
five  measures  for  courseware  effectiveness  and  in  relation  to  near-  and 
long-term  readiness  needs.  Based  on  this  examination,  we  have  out¬ 
lined  possible  ways  to  improve  development  of  the  IMI  courseware 
that  constitutes  the  core  of  the  current  TADLP  efforts,  and  also  taken 
a  broader  look  at  how  DL  could  be  improved  over  the  longer  term  to 
reach  larger  Army  readiness  goals. 


Condusions 

Our  assessment  of  TADLP’s  existing  program  for  developing  interac¬ 
tive  multimedia  instruction  (IMI)  courseware  (as  of  FY  2008)  led  to 
the  following  major  conclusions: 

•  TADLP  courseware  has  had  a  narrow  focus  that  limits  its 
potential. 

•  Our  assessment  showed  a  need  for  TADLP  improvement  with 
regard  to  all  measures  of  effectiveness  (program  impact,  efficiency, 
quality,  cycle  time,  and  responsiveness). 

•  TADLP  lacks  a  structured  process  for  evaluation,  assessment,  and 
improvement. 

•  The  potential  exists  to  significantly  expand  the  role  of  DL  in 
Army  training. 
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Recommendations 

We  then  identified  five  near-term  initiatives  that  would  increase  the 
impact  of  the  Army’s  current  IMI  program,  increase  the  quality  of  the 
product,  and  improve  the  efficiency  and  responsiveness  of  the  process. 
We  also  outlined  three  broader  options  for  increasing  the  program’s 
impact  and  improving  cost-effectiveness.  These  near-term  and  broader 
recommendations  are  summarized  in  Table  8.1. 


Table  8.1 

Summary  of  Near-Term  Initiatives  and  Broader  Options  Recommended 
in  This  Report 


Near-Term  Initiatives 

•  Add  flexibility  to  the  courseware  acquisition  strategy 

•  Ensure  sufficient  resources  per  training  module  for  stand-alone  IMI 

•  Undertake  systematic  process  improvements  to  reduce  IMI  cycle  times  for 
production 

•  Increase  local  participation  in  IMI  production  and  contract  administration 

•  Institute  a  program-level  IMI  evaluation  component  to  support  TADLP 
improvements 

Broader  Options  for  Increasing  Program  Impact  and  Cost-Effectiveness 

•  Employ  blended  learning  options  to  significantly  expand  DL's  role 

•  Integrate  TADLP  with  knowledge  management  (KM) 

•  Enhance  TADLP  management  functions 


Overall,  these  recommendations  constitute  a  case  for  significantly 
expanding  the  use  of  structured  and  unstructured  learning  to  enhance 
the  Army’s  training  and  leader  development  strategies.  While  the  direc¬ 
tions  recommended  here  represent  a  shift  from  some  current  practices, 
we  believe  such  changes  are  needed  for  DL  to  play  the  key  role  it  has 
been  assigned  in  leader  development  strategies  and  in  the  transforma¬ 
tion  of  training  in  the  Army. 
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Implications 

History  suggests  that  making  the  proposed  changes  will  not  be  easy 
and  will  require  more  than  the  details  discussed  in  this  report.  The 
issues  facing  the  Army’s  leadership  today  are  comparable  in  magnitude 
to  those  faced  in  the  post-Vietnam  era  and  the  shift  to  an  all-volunteer 
Army.  To  address  these  issues,  General  Abrams,  as  CSA,  initiated  a 
series  of  transformational  training  initiatives  that  eventually  resulted  in 
such  changes  as  the  movement  to  small-group  instruction  in  TRADOC 
schools  and  the  use  of  performance-based  collective  training  supported 
by  engagement  simulations  and  AAR  processes,  first  at  the  CTCs  and 
ultimately  throughout  the  Army.  While  the  shift  toward  DL  represents 
only  one  of  a  number  of  changes  to  training  needed  in  the  present  era, 
we  think  the  major  lesson  from  the  Abrams-initiated  training  trans¬ 
formation  still  applies:  the  key  components  to  effective  change  will 
be  top-down  command  emphasis,  encouragement  of  and  rewards  for 
change,  and  systematic  oversight,  supported  by  appropriate  measures 
of  effectiveness,  at  program  and  senior  leadership  levels. 
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In  this  appendix  we  describe  in  greater  detail  the  methods  used  to  con¬ 
duct  interviews  with  school  personnel.  We  have  also  included  copies  of 
the  preinterview  and  during-interview  questionnaires. 


DL  Survey  Part  1 

Because  schools  do  not  routinely  maintain  an  easily  accessible  list  of 
DL  courses,  we  compiled  a  draft  list  of  active  DL  courses  identified 
through  ATRRS,  which  was  sent  to  the  schools  to  use  as  a  base  case.1 
Additionally,  a  presurvey  asked  the  schools  to  verify  some  essential 
facts  about  their  active  courses.  Questions  in  the  presurvey  (included 
as  Part  1  of  the  questionnaire  in  this  appendix)  were  factual  in  nature 
and  sought  to  determine  the  number  of  academic  hours  per  course  by 
type  of  media,  who  paid  for  the  development,  how  the  product  was 
being  used,  the  degree  of  student-instructor  interaction,  and  whether 
typical  enrollees  belonged  to  the  AC  or  RC.2  Finally,  the  schools  were 
queried  as  to  the  completeness  of  the  course  list. 


1  The  draft  list  was  the  result  of  a  search  for  relevant  course  modules  within  ATRRS  (using 
“select  codes”),  reference  to  ATSC  administrative  data  that  indicated  a  module  was  embed¬ 
ded  inside  a  residential  course,  and  a  cross-check  with  administrative  data  kept  by  TOMA 
on  DL  courses. 

2  While  much  of  this  information  was  theoretically  available  via  the  course  administrative 
data  (CAD)  produced  for  the  course,  in  practice,  some  of  the  CAD  was  out  of  date.  Fur¬ 
ther,  because  there  is  no  unique  identifier  for  DL  courses  that  lasts  through  their  lifecycle, 
it  was  often  not  clear  whether  a  course  offered  in  ATRRS  was  one  developed  in  TADLP  or 
elsewhere. 
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DL  Survey  Part  2 

We  then  conducted  extended  structured  telephone  interviews  with 
officials  connected  with  20  Army  DL  programs  that  applied  for  and 
received  TADLP  funding  in  the  past.  Participants  were  solicited  via 
email  to  schools’  directors  and  key  staff  and  told  that  the  objective  was 
“to  identify  policy  options  for  making  improvements  in  The  Army  Dis¬ 
tributed  Learning  Program  (TADLP).”  Over  90  percent  of  the  schools 
approached  decided  to  participate  in  the  study. 

Each  interview  involved  1-8  participants  from  the  schools,  as  well 
as  2-5  personnel  from  RAND.  The  choice  of  participants  for  the  inter¬ 
view  was  left  to  the  school.  Participants  typically  included  contract¬ 
ing  representatives,  course  managers,  team  leads,  training  division  or 
branch  chiefs  responsible  for  the  production  of  DL,  and  in  some  cases 
the  school’s  director  of  training.  RAND  participants  always  included 
the  project  leader  and  at  least  one  other  study  member.  Most  often  3  or 
4  team  members  participated. 

Typically,  all  the  participants  gathered  for  one  session,  but  in  sev¬ 
eral  cases  RAND  conducted  separate  interviews  with  different  staff 
members  and  then  combined  the  results.  The  interview  was  adver¬ 
tised  as  a  60-  to  90-minute  event,  but  it  often  took  longer  in  practice, 
depending  on  the  interest  and  availability  of  school  participants. 

Interviews  consisted  of  open  discussion,  which  was  guided  by  a 
questionnaire  sent  prior  to  the  interview  (see  the  Part  1  questionnaire 
provided  later  in  this  appendix).  The  interviews  covered  a  wide  variety 
of  subjects  in  the  form  of  open-ended  questions.  To  encourage  candor, 
RAND  ensured  participants  and  schools  that  their  answers  were  fully 
confidential.  A  detailed  interview  protocol  appears  later  in  this  appen¬ 
dix  (Part  2  of  the  questionnaire).  Topics  addressed  during  the  interview 
included  the  following: 

•  The  role  of  DL  in  the  school’s  larger  training  strategy. 

•  How  training  content  to  be  converted  to  DL  was  selected  by  the 
schools. 

•  The  amount  of  resources  dedicated  to  the  DL  program  at  the 
school,  and  how  the  personnel  involved  were  organized. 
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•  School  efforts  to  assess  the  DL  products  they  produced. 

•  Obstacles  and  suggested  improvements  related  to  the  implemen¬ 
tation  of  TADLP  for  that  particular  school. 

The  protocol  was  sent  to  the  school  and  participants  several  days 
before  each  interview. 

Notes  from  each  interview  were  compiled  immediately  after  the 
interview  and  circulated  among  RAND  team  members  to  insure  accu¬ 
racy  and  completeness.  In  one  case  the  RAND  team  added  a  two-day 
site  visit  at  one  school  to  gain  more  detailed  information  and  to  verify 
some  of  the  conclusions. 
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RAND  Project  Survey  of  Army  Proponent  Schools 


Distributed  Learning 
Program 


Part  1:  Questions  to 
Be  Addressed  Prior  to 
Telephone  Interview 

TO  BE  COMPLETED  AND  RETURNED 
TO  mikes@rand.org  PRIOR  TO 
TELEPHONE  INTERVIEW 


PLEASE  FEEL  FREE  TO  CONTACT 
DR.  MICHAEL  SHANLEY 
IF  YOU  HAVE  ANY  QUESTIONS, 
EITHER  BY  EMAIL  AT  mikes@rand.org 
OR  BY  PHONE  AT  310-393-0411, 

EXT  7795 


Summer  2007 


RAND 


The  RAND  Corporation,  a  nonprofit  research  organization 
dedicated  to  improving  policy  and  decisionmaking 
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Overview  and  Purpose 

TRADOC  Headquarters  TDAD  is  sponsoring  The  RAND  Corpo¬ 
ration  (a  nonprofit  research  organization)  in  a  study  of  Distributed 
Learning  in  the  Army.  The  RAND  team  will  be  conducting  inter¬ 
views  during  the  summer  of  2007,  exploring  the  experience  and  plans 
of  proponent  schools  with  regards  to  DL.  A  particular  focus  is  struc¬ 
tured  training  that  is  key  to  readiness,  such  as  PME,  reclassification, 
and  selected  functional  courses.  TRADOC  intends  to  use  the  results 
to  formulate  policies  that  improve  The  Army’s  Distributed  Learning 
program  (TADLP). 

Instructions 

To  expedite  the  upcoming  telephone  interview,  please  answer  as  many 
of  the  following  questions  as  you  can  about  specific  DL  courses  for 
which  we  believe  you  are  the  proponent.  Any  remaining  questions  can 
be  addressed  during  the  telephone  interview,  or  beforehand  via  email 
or  phone. 

INSTRUCTIONS:  Use  your  mouse  to  click  the  space  or  type  in  other 
information  that  corresponds  to  your  answer  for  each  of  the  following 
questions.  All  question  answers  can  be  edited  if  you  want  to  change 
your  answer. 

Then  please  send  the  completed  questionnaire  back  as  a  Word  docu¬ 
ment  via  email. 

Thank  you  very  much  for  your  participation. 
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Questions  for  Following  Course: 

Number  =  (course  number),  Phase  =  (Number),  Name  =  (name  of  course) 


1.  Who  paid  to  have  the  DL  product  developed?  (Select  all  that 
apply) 

a.  _ HQTRADOC  (through  TADLP  contracting  process) 

b.  _ ARNG 

c.  _ Another  Army  organization  (list _ ) 

d.  _ Funded  internally  by  proponent 

e.  _ Other;  list _ 

2.  How  is  the  DL  product  used?  (Select  all  that  apply) 

a.  As  a  separate  course  phase  that  includes  passing  a  test  as  a 
graduation  requirement 

b.  _ As  a  separate  course  phase  that  does  not  include  a  separate  test 

c.  _ Available  for  refresher,  reachback,  or  sustainment  training 

d.  _ Available  for  self  development 

e.  _ Other;  list _ 

3.  What  degree  of  interaction  between  the  course  instructor  and 
the  student  is  envisioned  during  DL  training?  (Select  one) 

a.  _ Very  low  (questions  can  be  asked  at  student’s  initiative  through 

Army  Help  Desk) 

b.  _ Low  (an  instructor  assigned  to  the  course  can  be  contracted  by 

students) 

c.  _ Moderate  (some  student/instructor  interaction  is  built  into  the 

course) 

d.  _ High  (instructor-student  interaction  is  structured  into  course  as 

a  key  learning  vehicle 

e.  _ Other;  list _ 
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4.  What  is  the  maximum  delay  allowed  between  the  completion 
of  the  DL  phase  and  the  following  course  phase,  for  the  DL 
portion  to  count? 

a.  _ Not  relevant  (e.g.,  DL  not  a  separate  phase  of  the  course) 

b.  _ No  requirement 

c.  _ (List  number  of  days,  weeks,  months)  (underline  what  applies) 

5.  Total  academic  hours  of  DL  portion  or  phase  of  the  course? 

a. _ hours 

6.  What  are  academic  hours  by  type  of  media? 

a.  _ hours  of  IMI 

b.  _ hours  ofVTT 

c.  _ hours  of  Web-based,  instructor  led  training 

d.  _ hours  of  other  media;  list _ 

7.  What  is  the  component  of  typical  enrollees? 

a.  _ Mostly  RC 

b.  _ Mostly  AC 

c.  _ Significant  number  of  both 

d.  _ Other _ 
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RAND  Project  Survey  for  Distributed  Learning 


Distributed  Learning 
Experience  and  Issues 

Part  2:  Questions  to  be 
Addressed  in  Telephone 
Interview 

PLEASE  HAVE  ALL  PARTICIPANTS 
REVIEW  THIS  DOCUMENT  PRIOR  TO 
THE  TELEPHONE  INTERVIEW 


PLEASE  FEEL  FREE  TO  CONTACT 
DR.  MICHAEL  SHANLEY 
IF  YOU  HAVE  ANY  QUESTIONS, 
EITHER  BY  EMAIL  AT  mikes@rand.org 
OR  BY  PHONE  AT  310-393-0411, 

EXT  7795 


June  2007 


IM  The  RAND  Corporation,  a  nonprofit  research  organization 
dedicated  to  improving  policy  and  decisionmaking 


•  • 
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RAND’s  Assessment  of  Army  Distributed  Learning  (dL) 

•  Overview:  TRADOC  Headquarters  TDAD  is  sponsoring  The 
RAND  Corporation  (a  non-profit  research  organization)  in  a 
study  of  Distributed  Learning  in  the  Army.  The  RAND  team  will 
be  conducting  interviews  during  the  summer  of  2007,  exploring 
the  experience  and  plans  of  proponent  schools  with  regards  dL. 
A  particular  focus  is  structured  training  that  is  key  to  readiness, 
such  as  PME,  reclassification  and  selected  functional  courses. 
TRADOC  intends  to  use  the  results  to  formulate  policies  that 
improve  The  Army’s  Distributed  Learning  program  (TADLP). 

•  Purpose  of  interview:  As  part  of  our  assessment  of  dL,  we  are 
conducting  interviews  with  members  of  proponent  schools  who 
are  responsible  for  the  development  and  production  of  distributed 
learning  content.  We  are  interested  in  your  experiences  and  in 
what  you  have  found  to  be  enablers  for,  and  obstacles  to  distrib¬ 
uted  learning  developed  by  your  school.  We  are  also  interested  in 
your  opinions  about  how  to  make  improvements  in  the  program. 

•  RAND  participants:  Drs.  Michael  Shanley  and  other  members 
of  the  project  team  will  be  leading  the  interviews.  A  research 
assistant  may  also  be  present  for  the  purpose  of  recording  accurate 
notes. 
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Informed  Consent  Information 


RAND  Consent  Language 

•  RAND  will  use  the  information  you  provide  for  research  pur¬ 
poses  only,  and  will  not  disclose  your  identity  or  information  that 
identifies  you  to  anyone  outside  of  the  research  project,  except  as 
required  by  law  or  with  your  permission. 

•  No  one,  except  the  RAND  research  team,  will  have  access  to  the 
information  you  provide.  RAND  will  only  produce  summary 
information  from  our  collective  set  of  interviews. 

•  We  will  destroy  all  information  that  identifies  you  after  the  study 
has  concluded. 

•  You  do  not  have  to  participate  in  the  interview,  and  you  can  stop 
at  any  time  for  any  reason. 

•  Your  participation  or  nonparticipation  will  not  be  reported  to 
anyone. 

•  You  should  feel  free  to  decline  to  discuss  any  topic  that  we  raise. 

•  Do  you  have  any  questions  about  the  study? 

•  Do  you  agree  to  participate  in  the  interview? 

If  you  have  any  specific  questions  about  this  about  this  research,  you 
may  contact: 


Michael  Shanley,  Ph.D. 

Policy  Researcher  &  Principal 
Investigator 
RAND 
P.O.  Box  2138 

Santa  Monica  CA  90407-2138 
Telephone:  310-393-0411,  x7795 
Email:  mikes@rand.org 


Tora  K.  Bikson,  Ph.D. 

Human  Subjects  Protection 
Committee 
RAND 
P.O.  Box  2138 

Santa  Monica  CA  90407-2138 
Telephone:  310-393-0411 
FAX:  310-393-4818 
Email:  Tora_Bikson@rand.org 
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Questions  for  Proponents  on  Their  Distributed 
Learning  Programs 

For  purposes  of  the  following  questions,  distributed  learning  (dL)  is 
defined  as  the  use  of  technology  to  deliver  training  without  the  need 
for  a  collocated  instructor.  If  an  answer  varies  by  dL  area,  the  focus  of 
these  questions  is  intended  to  be  on  structured  training  that  is  key  to 
readiness  (i.e.,  training  for  reclassification,  BNCOC,  ANCOC,  CCC, 
or  key  functional  courses).  In  the  Pre-survey  you  should  have  received, 
we  listed  courses  or  course  phases  that  fall  into  this  category  for  which 
we  believe  your  school  is  the  proponent. 


A.  The  role  of  dL  in  your  school’s  overall  training  strategy 

1.  What  types  of  dL  do  you  consider  key  to  your  school’s  training 
strategy? 

_ IMI 

_ VTT 

_ Other  collaborative  technologies 

_ web-based,  instructor  led 

_ Other  (list _ ) 

2.  What  is  your  school’s  general  strategy  or  concept  for  using  dL 
to  support  key  structured  training  as  described  above?  (If  relevant, 
describe  for  each  type  in  Question  1). 

_ Increase  the  number  of  tasks  that  can  be  trained  within  the 

institution 

_ Provide  knowledge-based  training  prior  to  residential  course 

phases 

_ Support  the  instructor  during  residential  training 

_ Provide  sustainment  training 

_ Other  (specify _ ) 
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3.  In  general,  what  other  benefits  do  you  see  that  dL  might  have  in 
supporting  key  structured  training  in  your  school?  (Please  explain 
answers  and  indicate  whether  you  believe  any  of  these  benefits  are 
already  being  realized  with  current  dL  courses.) 


Allows  delivery  of  more  institutional  training  away  from 
schoolhouse 

Reduces  average  learning  time  or  increases  training  quality 

Provides  sustainment  training 

Effective  for  training  with  following  characteristics 

(specify _ ) 

Other  (specify _ ) 


B.  dL  selection  process 

4.  For  structured  training  that  is  key  to  readiness,  what  is  the  selec¬ 
tion  process  for: 

□  choosing  training  to  be  converted  to  dL? 

□  choosing  the  approach  and  media  used? 

5.  Do  you  use  any  tools/decision  aids  to  support  the  process  of  choos¬ 
ing  training  to  convert  to  dL,  or  in  choosing  the  preferred  approach 
and  media?  Explain. 

6.  What  factors  are  considered  in  determining  what  institutional 
training  is  converted  to  IMI  within  your  school?  Explain  as  many 
criteria  you  are  aware  of. 

_ Army  or  TRADOC  headquarters  directives 

_ School  commandant  priorities 

_ Importance  of  the  instruction 

_ Suitability  of  the  instruction  for  IMI 

_ Number  of  soldiers  or  leaders  needing  the  training 

_ Other  (list) _ 
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C.  dL  resources 

7.  What  is  the  approximate  size  of  the  staff  that  plans,  coordinates, 
and  oversees  the  IMI  portion  of  your  dL  program  (including  IMI 
produced  for  use  in  residential  courses)?  Please  distinguish  full  and 
part  time  staff  and  describe  the  overall  organization  within  which  the 
staff  operates. 


8.  Are  you  adequately  resourced  to  develop  and  deliver  IMI?  Please 
explain. 


9.  What  additional  training  programs  (if  any)  do  you  see  as  necessary 
for  your  dL  staff? 


10.  Do  you  reuse  or  see  the  potential  for  reusing  any  of  your  school’s 
(or  another  school’s)  IMI  materials  in  other  courses  or  phases?  Please 
explain. 


D.  dL  assessment  program 

11.  What  methods  does  your  school  use  (or  plan  to  use)  to  assess  the 
quality  of  the  IMI  training  or  validate  course  tests?  Please  describe 
each  and  indicate  any  available  reports. 

_ Autogen  surveys 

_ End  of  course  opinion  surveys 

_ Knowledge/skill  tests  during  or  at  end  of  dL  training 

_  Other  _ 
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E.  Obstacles  and  suggested  improvements  in  TADLP  processes 
for  producing  dL 

12.  How  satisfied  or  dissatisfied  are  you  with  your  school’s  current 
IMI  development  program?  Please  rate  on  a  5  point  scale,  entering  an 
x  in  appropriate  box  for  each  row. 


I.Very 

satisfied 

2. Somewhat 
satisfied 

3.  Both 
satisfied  and 
dissatisfied 

4. Somewhat 
dissatisfied 

5.  Very 
dissatisfied 

Amount  of  content 
produced 

Length  of  time  to 
develop 

Efficiency  of  dL 
production 

Quality  of  dL 
products 

12a.  If  appropriate,  answer  same  questions  for  other  types  of  dL  (e.g., 
VTT,  collaborative  technologies,  web-based-instructor-led) 


13.  From  your  school’s  point  of  view,  in  what  areas  do  you  see  obsta¬ 
cles  or  challenges  for  implementing  a  dL  strategy  through  TADLP? 
Please  explain  in  each  area. 

_ Contracting  procedures,  (please  specify  contract:  DLXXI  or 

DLETP) 

_ Meeting  GFI  requirements 

_ Providing  SME  support 

_ Accomplishing  course  maintenance 

_ Adequacy  of  contractors 

_ Training  of  dL  staff  at  the  school 

_ Technology  required  (e.g.,  LMS  issues) 

_ Incentives  for  schools  (e.g.,  dL  reduces  school  resources) 

_ Cultural  issues  (e.g.,  belief  that  dL  cannot  be  effective) 

_ Other  processes  or  policies  (specify) 

_ Other  resource  issues  (e.g.,  size  of  dL  staff) 

_ Other  (list) _ 
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14.  Considering  all  the  obstacles  noted,  which  one(s)  do  you  consider 
to  be  the  most  important? 


15.  What  changes  or  improvements  would  you  suggest  to  address 
each  obstacle? 


16.  What  do  you  consider  to  be  the  most  important  improvement 
that  needs  to  be  made  to  TADLP?  Please  explain. 


17.  Considering  existing  dL  policies  and  standards  and  support  from 
HQTRADOC,  in  what  areas  would  you  want  more  guidance  or 
support?  In  what  areas  would  you  want  more  flexibility? 


F.  Further  explanation  of  data  in,  or  not  in,  ATRRS 

18.  From  ATRRS,  we  noticed  a  number  of  dL  courses  that  were 
fielded  under  TADLP  in  the  last  few  years,  but  did  not  appear  to  be 
currently  in  use.  If  this  applies  to  your  school,  what  factors  contribute 
to  the  non-use  of  completed  dL  courses  (indicate  all  that  apply)? 

_ Command  decision  or  course  design  change 

_ Content  obsolete 

_ Maintenance  needed 

_ Courses  not  removed — rather,  still  under  development 

_ Other  (specify _ ) 

19.  ATRRS  also  shows  there  are  many  more  enrollees  than  graduates 
in  dL  courses.  If  this  applies  to  your  dL  courses,  what  do  you  believe 
is  the  reason  for  the  low  graduation  rate? 
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20.  Is  there  substantial  use  of  your  school’s  IMI  material  outside 
of  structured  training  (e.g.,  for  self  development)  that  does  not  get 
recorded  on  ATRRS?  Is  there  any  record  of  this  use  at  school  level? 


21.  Are  there  any  dL  course  phases  produced  by  your  school  that  you 
consider  key  to  readiness  but  that  were  not  in  the  Pre-survey?  If  yes, 
please  describe,  and,  if  possible,  list  information  below  or  in  separate 
email. 

Course  number  Phase  Course  name  Academic  Hrs  Media  used 

a.  _ 

b.  _ 

c.  _ 


22.  Do  you  embed  dL  material  within  residential  courses?  If  yes, 
please  list  relevant  courses  and  other  information  below  or  in  separate 
email. 

Course  number  Course  name  DL  Academic  Hrs  Media  used 

a.  _ 

b.  _ 

c.  _ 


G.  Conclusions 

23.  Do  you  have  any  additional  suggestions  on  how  to  improve  the 
management,  resourcing,  or  processing  of  dL  courseware  through  the 
TADLP? 


24.  Overall,  what  is  the  most  important  lesson  you  have  learned 
about  dL? 


APPENDIX  B 


Potential  Explanations  for  Outcomes  from  Army 
Schools'  Point  of  View 


In  this  appendix  we  discuss  the  results  of  interviews  with  training 
development  personnel  from  20  different  Army  schools  and  programs. 
These  interviews  focused  on  the  proponent  schools’  assessment  of  the 
obstacles  and  challenges  facing  TADLP.  This  information  was  used  as 
one  input  for  recommended  improvements  presented  in  Chapter  Four. 


Obstacles  to  the  DL  Program 

Army  proponent  schools  identified  a  comprehensive  list  of  obstacles 
they  saw  in  the  DL  program,  which  we  grouped  into  six  categories  as 
shown  in  the  left-hand  column  of  Table  B.l.  The  categories  are  ordered 
according  to  the  percentage  of  schools  that  identified  a  “key  issue”  (i.e., 
major  obstacle  or  challenge)  in  that  category,  as  shown  in  the  middle 
column.  The  percentage  of  schools  mentioning  any  issue  within  a 
particular  category  (including  those  of  secondary  importance)  is  also 
shown  in  the  middle  column  of  the  table. 

Results  show  that  the  schools  found  a  wide  range  of  obstacles 
within  the  DL  program;  note,  for  example,  that  all  six  categories  in  the 
table  were  noted  by  more  than  half  the  schools.  In  addition,  the  most 
critical  category  of  obstacle  for  the  schools  involved  the  inadequate  pro¬ 
vision  of  resources  and  resource  mechanisms  to  develop,  deliver,  and 
evaluate  training.  All  schools  mentioned  some  obstacle  in  this  area, 
and  60  percent  identified  one  or  more  key  issues  in  this  category.  The 
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Table  B.1 

Obstacles  to  TADLP 


Obstacle 

Percent 

Key  Issue 

Percent 

Mentioned 

Resource  adequacy  and  mechanisms 

60 

100 

Courseware  development  processes 

30 

90 

Courseware  maintenance  practices 

30 

85 

School-contractor  interaction 

30 

60 

Technology/IT/standards  issues 

25 

85 

Key  stakeholders  buy-in 

5 

55 

remaining  categories  were,  except  for  the  last,  given  about  equal  weight 
in  terms  of  how  frequently  key  obstacles  were  noted  in  these  categories. 

The  table  above  was  derived  from  a  telephone-based  survey  with 
representatives  from  20  proponent  schools.  We  asked  participants: 

•  “From  your  school’s  point  of  view,  in  what  areas  do  you  see 
obstacles  or  challenges  for  implementing  a  DL  strategy  through 
TADLP?  Please  explain  in  each  area.” 

After  discussing  all  the  challenges  mentioned  by  a  school,  we  asked 
schools: 

•  “Considering  all  the  obstacles  noted,  which  one(s)  do  you  con¬ 
sider  to  be  the  most  important?” 

Some  schools  identified  only  one  issue  as  the  “most  impor¬ 
tant,”  while  other  schools  said  they  could  not  identify  a  single  “most 
important”  issue  but  instead  considered  multiple  issues  to  be  equally 
important. 

In  total,  we  received  52  nominations  of  “most  important”  issues 
from  the  20  schools  surveyed.  In  the  following  discussion,  we  refer  col¬ 
lectively  to  these  responses  as  “key  issues.”  We  then  sorted  the  obstacles 
into  the  six  overarching  categories  shown  above.  We  will  discuss  each 
of  the  six  categories. 
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Resource  Issues 

Overall,  resource  issues  were  cited  as  a  challenge  by  100  percent  of  the 
schools  interviewed  and  were  identified  as  a  key  issue  by  60  percent 
of  the  schools,  more  than  twice  the  percentage  for  any  other  category. 

The  most  commonly  cited  resource  issue  was  “a  lack  of  sufficient 
or  qualified  SME  support,”  either  from  the  school  side,  the  contractor 
side,  or  both.  Seventy  percent  of  the  schools  described  this  issue  as  an 
obstacle,  and  40  percent  cited  it  as  a  key  issue.  While  support  for  SMEs 
was  commonly  funded  in  the  award  to  contractors,  according  to  the 
schools,  many  contractors  were  unable  to  find  SMEs  that  were  suffi¬ 
ciently  qualified  to  meet  school  expectations. 

In  addition  to  general  SME  support,  45  percent  of  the  schools 
noted  specific  DL-related  tasks  for  which  TADLP  did  not  provide 
funds,  with  10  percent  noting  these  unfunded  tasks  as  a  key  issue. 
Respondents  stated  that  funding  was  lacking  in  several  areas,  including: 

•  Completing  up-to-date  GFI. 

•  Undertaking  front-end  analyses  for  properly  specifying  training 
requirements. 

•  Providing  student  support  (both  technical  and  substantive)  suf¬ 
ficient  to  ensure  high  participation  in  the  program. 

•  Providing  a  school  capability  for  minor  maintenance  of  course¬ 
ware  or  the  creation  of  simple  DL  products  that  require  rapid 
distribution. 

•  Providing  for  assessment  of  the  outcomes  of  DL  training. 

A  significant  number  of  schools  also  expressed  confusion  or 
doubt  about  how  or  whether  “alternative”  approaches  to  DL  would 
or  could  be  funded  under  the  current  program.  Sixty-five  percent  of 
the  schools  mentioned  this  issue,  and  15  percent  mentioned  it  as  a  key 
issue.  For  example,  despite  the  fact  that  ATSC  is  currently  developing 
a  blended  learning  model  to  use  within  TADLP,  schools  perceived  that 
TRADOC  might  not  provide  sufficient  funds  to  develop  IMI  content 
to  teach  more  complex  learning  goals  and  for  supporting  instructor- 
led,  web-based  instruction. 
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In  general,  the  schools  expressed  a  desire  to  use  DL  as  a  way  not 
only  to  provide  knowledge-level  learning  (as  do  current  courses),  but 
also  to  support  higher  levels  of  learning,  such  as  critical  thinking  and 
problem-solving  skills.  Further,  schools  do  not  feel  that  the  current  pro¬ 
gram  provides  sufficient  funding  to  support  a  collaborative  approach  to 
training. 

In  explaining  the  problem,  some  schools  also  mentioned  related 
obstacles,  such  as  the  insufficiency  of  the  Army’s  Saba  Learning  Man¬ 
agement  System,  as  currently  designed,  to  support  anything  beyond 
a  knowledge-based,  stand-alone  approach.  Other  schools,  citing  their 
early  experience  with  TADLP,  expressed  concern  that  bringing  new, 
innovative  efforts  to  TADLP  would  somehow  lead  to  a  financial  pen¬ 
alty  for  their  school. 

Courseware  Development  Process  Issues 

Courseware  development  processes  cover  administrative  procedures 
and  management  for  the: 

•  submission  of  the  application  for  a  DL  course, 

•  selection  process  for  DL  courses, 

•  development  of  the  delivery  order  once  a  course  is  selected, 

•  development  of  the  courseware  over  the  period  of  the  contract, 

and 

•  testing  of  the  content  for  playability  with  SCORM  and  Saba. 

Almost  universally  (90  percent),  contractors  cited  some  kind  of 
obstacle  in  this  area.  However,  only  30  percent  of  schools  cited  the 
courseware  development  process  as  a  key  obstacle  to  DL  success.  None¬ 
theless,  the  schools  did  consider  this  an  important  issue.  Evidence  for 
this  comes  from  another  part  of  the  survey,  where  schools  were  asked 
about  satisfaction  with  “length  of  time  to  develop”  within  TADLP. 
Sixty-seven  percent  rated  themselves  as  “very  dissatisfied,”  and  another 
19  percent  rated  themselves  as  “somewhat  dissatisfied.” 

Schools  provided  a  number  of  explanations  for  the  obstacles  they 
described.  Nearly  all  schools  stated  that  the  entire  process  was  too  slow 
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overall.  However,  some  also  pointed  to  particular  areas  such  as  the  time 
it  took  to 

•  turn  an  idea  into  a  contract  at  the  front  end  of  the  process,  or 

•  test  at  the  back-end  of  the  process. 

The  time  required  for  back-end  testing  was  particularly  frustrat¬ 
ing  to  representatives  of  at  least  one  of  the  schools,  given  how  much 
faster  a  similar  process  could  be  achieved  in  the  academic  sector.1 

A  number  of  schools  also  stated  that  the  overall  process  has  too 
many  steps  and  too  much  paperwork,  and  changes  too  much  over 
time.  Some  schools  simply  noted  the  additional  workload  associated 
with  a  development  process  in  which  requirements  frequently  change 
after  the  delivery  order  is  signed.  Other  schools  noted  frequent,  often 
confusing,  changes  in  procedures  and  templates  by  ATSC,  without 
perceived  justification  or  apparent  benefit.  Finally,  a  number  of  schools 
mentioned  the  considerable  effort  expended  to  submit  an  application 
for  DL  funding,  with  (at  least  in  some  cases)  only  a  small  probability  of 
success.  One  school  said  they  had  recently  submitted  seven  applications 
without  receiving  an  award.  Along  similar  lines,  a  couple  of  schools 
expressed  frustration  that  TRADOC  did  not  provide  more  guidance 
up  front  as  to  what  it  was  willing  to  fund  so  that  schools  could  more 
effectively  decide  where  to  apply  their  efforts. 

Many  schools  also  described  the  current  process  as  too  bureau¬ 
cratic  and  inflexible  to  meet  school  needs,  especially  with  regard  to  GFI 
requirements,  the  application  of  SCORM  even  when  the  likelihood  of 
reuse  is  small,  and  the  procedures  for  obtaining  maintenance  funding. 
A  couple  of  schools  also  criticized  the  lack  of  a  procedure  to  support 
faster  courseware  production  when  needed  for  urgent  or  high-priority 
projects.  Schools  reported  that  waivers  from  normal  procedures  are 
time-consuming  and  difficult  to  obtain  because  of  “the  large  number 
of  people  in  different  places  that  have  to  agree.” 


1  ATSC  has  undertaken  a  number  of  initiatives  to  shorten  the  time  required  for  back-end 
testing.  While  a  backlog  was  eliminated  after  one  study,  difficult  new  testing  requirements 
led  to  its  reemergence. 
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Finally,  some  schools  felt  that  training  content  suffered  from  the 
lack  of  “someone  in  charge”  of  courses  overall.  In  the  development 
of  several  courses,  a  lack  of  continuity  of  the  project  team  was  noted, 
within  both  the  school  and  the  contractor.  In  another  case,  a  school 
noted  that  multiple  organizations  had  some  responsibility  for  getting 
a  course  “up  and  running,”  but  no  one  had  responsibility  for  follow¬ 
ing  through  to  ensure  all  the  steps  were  cost-effectively  completed  and 
integrated. 

Issues  Concerning  the  Maintenance  of  IMI 

Another  challenge  to  DL  program  implementation  concerns  the  dif¬ 
ficulty  of  implementing  timely  changes  to  courseware  after  the  devel¬ 
opment  effort  is  completed.  “Maintenance”  of  IMI  software  in  this 
context  implies  minor-to-medium  changes  in  a  course,  as  opposed  to 
a  complete  overhaul.  The  need  for  maintenance  can  come  from  several 
sources — it  could  be  making  minor  changes  to  the  entire  course  to 
keep  it  up  to  date,  upgrading  the  course  through  a  new  delivery  tech¬ 
nology,  or  replacing  or  adding  a  relatively  small  section  of  new  content 
to  the  course  (perhaps  due  to  a  change  in  doctrine  or  requirements,  or 
due  to  an  assessment  that  the  current  lessons  are  not  working  well). 

In  the  commercial  IMI  market,  there  is  an  underlying  assump¬ 
tion  that  clients  will  want  to  update  content  fairly  often  and  that  such 
“maintenance”  will  be  part  of  the  planning  to  support  the  product 
once  it  is  delivered.  Such  changes  are  handled  either  through  a  main¬ 
tenance  contract,  which  often  specifies  the  hours  of  programmer  time 
monthly  that  are  included  in  the  maintenance  contract.  If  the  client 
needs  to  go  beyond  that  base  number  of  hours  in  a  month,  then  a  sepa¬ 
rate  small  contract2  is  let.  An  alternative  is  to  have  maintenance  “on 
call”  as  needed  with  an  hourly  rate  paid  to  the  developer. 

The  current  TRADOC  process  for  implementing  postproduction 
maintenance  of  courseware  often  requires  a  new  acquisition  cycle  and, 
as  a  result,  could  take  several  years  from  the  identification  of  the  need 
to  the  fielding  of  the  updated  content. 


2  Often  a  simple  agreement  written  in  regularly  used  contract  language. 
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Nearly  all  schools  (85  percent)  cited  maintenance  as  an  obstacle, 
and  30  percent  noted  it  as  a  key  obstacle  to  providing  high-quality 
courseware.  In  fact,  schools  that  did  not  see  maintenance  as  an  issue 
either  had  no  active  courses  or  had  found  a  way  to  update  without 
going  through  the  formal  process  of  applying  for  funds.  In  explain¬ 
ing  their  concerns,  schools  noted  that  DL  products  need  immediate 
and  continual  change  to  remain  relevant,  much  in  the  same  way  that 
residential  courses  need  updating  on  a  continual  basis.  Requirements 
for  change  can  be  minimized  through  the  careful  choice  of  content 
for  DL.  However,  less  and  less  of  the  Army’s  training  content  remains 
stable.  In  addition,  the  training  development  staff  noted  that  instruc¬ 
tional  departments  were  reluctant  to  “own”  a  course  for  which  they 
could  not  control  the  content,  i.e.,  keep  up  to  date  via  maintenance.  No 
instructor  wants  to  be  held  responsible  for  courses  that  are  inaccurate 
or  out  of  date. 

The  need  for  change  has  led  some  schools  to  create  an  internal 
capability  to  make  small  maintenance  changes  in  courseware  them¬ 
selves.  However,  schools  have  limited  ability  to  make  even  minor 
changes  because  they  lack  the  expertise  to  make  the  changes,  or  enough 
familiarity  with  contractors’  hie  structure  and  authoring  tools  to  prop¬ 
erly  perform  the  maintenance. 

Moreover,  the  problem  is  compounded  if  a  school  has  multiple 
courses  that  have  been  developed  by  different  contractors,  a  likely  out¬ 
come  given  the  current  contractor  bidding  process.  When  each  piece  of 
content  is  completed  by  a  separate  contractor,  in-house  staff  typically 
have  to  gain  expertise  with  multiple  new  hie  structures  and  author¬ 
ing  tools.  This  problem  has  led  at  least  one  school  to  standardize  the 
authoring  and  content  environment  for  all  its  courses  (requiring  con¬ 
tractors  to  use  a  specihc  hie  structure  and  set  of  authoring  tools)  so  that 
it  might  train  its  staff  to  make  changes  in  DL  courses. 

In  practice,  the  schools  noted  that  the  disconnect  between  the 
need  for  change  and  the  current  processes  for  maintenance  often  leads 
schools  to  declare  a  course  obsolete  rather  than  submit  a  proposal  for 
change.  Thus,  some  DL  courses  ended  up  having  artificially  short 
lifecycles. 
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School-Contractor  Interaction  Issues 

Another  category  of  obstacles  to  DL  program  implementation  concerns 
problems  in  the  working  relationship  between  schools  and  contractors. 
Schools  cited  this  issue  60  percent  of  the  time,  and  cited  it  as  a  key  issue 
30  percent  of  the  time  (the  same  as  for  the  last  two  categories).  Thus, 
while  relatively  fewer  schools  cited  this  category  as  an  obstacle  (other 
categories  were  mentioned  85  percent  or  more  of  the  time),  the  issue 
was  more  likely  to  be  a  “key”  one  if  mentioned  at  all. 

In  explaining  this  issue,  some  schools  reported  that  there  was 
high  variability  in  the  quality  of  the  contractors.3  According  to  some 
schools,  some  contractors  simply  did  not  understand  the  school’s  sub¬ 
ject  area  well  enough  to  produce  quality  content.4  In  addition,  schools 
expressed  frustration  in  not  being  able  to  prescreen  contractors  in  order 
to  avoid  such  a  mismatch.  In  cases  where  the  contractors  would  need 
to  learn  a  great  deal  before  being  able  to  develop  specific  content,  the 
schools  felt  there  was  an  additional  burden  placed  on  them  to  edu¬ 
cate  the  contractors.  Schools  also  felt  that  there  were  no  incentives  in 
the  program  to  reward  improvement  in  performance  over  time — for 
example,  to  reward  good  performers  with  follow-on  work.  Similarly, 
some  schools  felt  there  was  no  way  to  impose  consequences  for  poor 
performance.  For  example,  schools  felt  that  contractors  experienced  no 
consequences  for  lateness  or  poor-quality  output.5 

The  focus  group  with  contractors  documented  some  of  their  frus¬ 
trations  with  current  processes.  These  included  schools  that  expected 
significant  mid-project  changes  in  course  design  or  content6  but  no 
change  to  the  contract  or  increase  in  compensation.  Contractors  also 


3  At  least  from  the  point  of  view  of  being  able  to  produce  quality  content  in  the  particular 
school’s  occupational  area. 

4  This  was  reportedly  due  to  getting  SMEs  later  in  the  development  process,  the  quality  of 
some  SMEs,  and  a  lack  of  regular  collaboration  between  developer  and  school  representatives. 

5  ATSC  is  currently  working  on  a  system  to  get  more  feedback  from  schools  regarding  con¬ 
tractor  performance. 

6  This  reportedly  occurred  frequently  when  points  of  contact  at  the  school  changed  and 
the  new  point  of  contact  had  a  perspective  on  content  or  instructional  methods  that  differed 
from  the  predecessor’s. 
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cited  the  high  turnover  of  school  personnel  working  on  the  project, 
as  well  as  inadequate  GFI  and  slow  review  of  content.  In  a  few  cases 
contractors  also  reported  a  combative  attitude  by  the  school  toward  the 
contractor  (e.g.,  seeming  to  assume  the  contractor  was  trying  to  maxi¬ 
mize  profit  at  the  expense  of  the  output).  Moreover,  contractors  saw  no 
consequences  to  the  school  for  changing  direction  and,  importantly, 
no  way  to  effect  changes  in  the  contract  when  the  need  for  additional 
work  was  identified.  Contractors  also  noted  the  lack  of  any  incentive 
for  follow-on  work  for  those  who  perform  well.  In  their  view,  there  was 
little  to  no  effort  made  to  build  a  solid  working  relationship  with  the 
client  that  could  be  amortized  across  multiple  contracts. 

Technology/IT/Standards  Issues 

Another  category  of  challenges  to  DL  program  implementation  con¬ 
cerns  obstacles  related  to  the  technologies  and  IT  standards  (including 
those  related  to  SCORM  and  the  ALMS)  that  surround  IMI.  Schools 
cited  this  issue  85  percent  of  the  time,  but  cited  it  as  a  key  issue  only 
about  25  percent  of  the  time. 

While  most  schools  acknowledged  ongoing  improvements  within 
the  Saba  system,  they  had  a  variety  of  comments  on  their  experience 
to  date.  Some  thought  the  Saba  LMS  was  too  narrow  in  the  type  of 
DL  it  would  support,  namely,  the  asynchronous  IMI  model  now  used 
by  the  Army  for  structured  courses.  Others  thought  that  it  had  aimed 
at  too  broad  an  array  of  functions  (e.g.,  career  management,  content 
management,  administrative  management,  tracking  of  tasks  trained 
to  a  detailed  level)  and  hence  increased  the  risk  of  failure,  made  inte¬ 
gration  difficult,  and  delayed  the  fielding  of  some  of  the  more  basic 
capabilities.  Most  schools  thought  the  Saba  product  was  not  particu¬ 
larly  “user-friendly,”  at  least  with  regard  to  the  functionality  they  were 
interested  in  at  the  local  level.  Respondents  also  associated  the  Army’s 
product  with  long  delays  in  courseware  fielding,  as  the  courses  could 
remain  in  the  ALMS  testing  queue  for  many  months  and,  once  tested, 
would  not  pass  because  the  courses  were  not  developed  to  the  most 
recent  ALMS  standard.  Schools  also  reported  that  some  students  had 
experienced  difficulty  getting  credit  for  courses  due  to  technical  issues 
with  the  ALMS. 
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Other  technology-related  problems  cited  by  schools  included  the 
complex  authoring  tool  environment  that  makes  it  difficult  to  train 
staff  to  make  simple  changes  and  updates  to  finished  IMI  products. 
Schools  also  mentioned  the  lack  of  tools  to  support  rapid  development 
when  that  function  is  needed,  as  well  as  problems  with  playability  of 
content  on  selected  posts  with  CONUS.  In  particular,  for  security  rea¬ 
sons,  the  Directorate  of  Information  Management  (DOIM)  on  selected 
posts  would  not  allow  all  the  software  tools  needed  to  allow  content  to 
run  within  their  network. 

Finally,  schools  (and  TRADOC/ATSC  headquarters  staff  as  well) 
noted  the  high  upfront  investment  required  to  build  in  the  SCORM 
standards  associated  with  courseware  initially,  and  then  to  update  the 
courseware  as  the  SCORM  standards  changed  over  time.  The  goal  of 
SCORM  standards  is  to  allow  content  sharing  across  LMS,  and  to  sup¬ 
port  the  possibility  of  reuse  in  future  development  efforts.  However, 
schools  found  that  the  SCORM  standards  were  not  yet  mature  enough 
to  ensure  that  training  content  that  worked  on  one  LMS  would  neces¬ 
sarily  play  correctly  on  another  commercial  LMS;  they  also  pointed 
out  that  reuse  was  rarely  needed.  At  the  same  time,  the  schools  saw  the 
standards  as  leading  to  long  delays  in  courseware  fielding  and  compli¬ 
cating  the  process  of  changing  and  updating  courseware. 

Issues  Related  to  Buy-In  from  Key  Stakeholders 

The  final  category  of  obstacles  concerns  the  level  of  commitment  or 
“buy-in”  among  key  Army  stakeholders.  This  issue  tended  to  involve 
senior  leadership,  school  leadership,  unit  leadership,  and  the  students 
themselves.  Some  stakeholder  reluctance  was  due  to  their  beliefs  about 
DL,  and  some  came  from  direct  experience.  Buy-in  was  least  frequently 
noted  as  a  key  issue  by  the  schools  (5  percent)  but  was  cited  in  some 
form  by  over  half  (55  percent)  of  the  schools.  While  most  of  the  schools 
acknowledged  that  stakeholder  buy-in  has  improved  since  the  early 
years  of  DL,  they  emphasized  that  many  stakeholders  were  still  reluc¬ 
tant  to  fully  support  DL. 

At  the  schools,  some  stakeholders  did  not  believe  in  the  effec¬ 
tiveness  of  DL  as  a  modality  for  training  and,  as  a  result,  sought  to 
resist  its  expansion.  Specific  issues  mentioned  ranged  from  a  lack  of 
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training  transfer  to  the  inability  to  prevent  cheating  on  online  tests. 
Schools  reported  that  unit  commanders  expected  their  soldiers  to  com¬ 
plete  their  DL  on  their  own  time,  thereby  implying  that  it  is  a  relatively 
unimportant  component  of  training.  The  need  to  work  in  off-duty 
hours  and  concerns  about  quality  and  technical  complexity  have  led 
some  students  to  develop  an  unfavorable  view  toward  DL,  and  many 
have  chosen  traditional  training  when  that  option  has  been  available. 

Within  proponent  schools,  DL  has  been  viewed  more  as  a 
TRADOC  Headquarters  strategy  for  reducing  school  resources 
than  as  a  method  to  transform  training.  In  particular,  some  believe 
that  resourcing  methodologies  for  DL  leave  schools  with  inadequate 
resources  to  support  the  training.  While  at  least  some  resourcing  prac¬ 
tices  have  been  updated  in  recent  years  (and  higher  resourcing  for  DL 
is  sometimes  a  possibility),  some  schools  still  believe  that  innovative 
efforts  in  the  area  of  DL  will  lead  to  a  resource  penalty. 

Much  more  could  be  ascertained  concerning  stakeholder  buy-in 
through  a  survey  of  students,  teaching  departments  and  DL  instruc¬ 
tors.  This  is  an  area  for  further  research. 


Bibliography 


ARI,  Training  for  Future  Operations — Digital  Leaders’  Transformation  Insights,  ARI 
Special  Report  No.  53,  November  2002. 

- ,  Information  Paper,  2004.  As  of  December  21,  2007: 

http://www.hqda.army.mil/ari/surveys/ 

- ,  Distance  Learning  Findings,  Fall 2006  Sample  Survey  of  Military 

Personnel,  March  27,  2007. 

Army  Chief  of  Staff  Memorandum,  Army  Training  and  Leader  Development 
Guidance,  August  2008. 

ASTD  Certification  Institute,  E-Learning  Courseware  Certification  (ECC) 
Standards  (1.5),  Alexandria,  Va.:  American  Society  for  Training  and  Development, 
2001-2003. 

BAE  Systems,  Executive  Summary  Knowledge  and  Information  Management  in  the 
Generating  Force,  December  2007. 

Bloom,  B.  S.,  et  ah,  Taxonomy  of  Educational  Objectives  (two  volumes:  The  Affective 
Domain  and  The  Cognitive  Domairi),  New  York:  David  McKay,  1964.  As  of 
January  2011: 

http://www.nwhnk.eom/.-donclark/hrd/bloom.html 

Brandon  Hall  Research  Staff,  Flow  to  Buy  E-Learning  Systems,  Tools,  and  Services, 
Sunnyvale,  Calif.:  Brandon  Hall  Research,  Woodhill,  2006. 

Chapman,  B.,  “Optimizing  Learning  Strategies”  Workshop  Series,  2008.  Two- 
day  workshop  for  learning  professionals.  Workshop  Materials  available  through 
Chapman  Alliance  LLC,  Salt  Lake  City,  Utah.  As  of  January  2011: 
http://www.chapmanalliance.com 

Chapman,  B.,  et  ah,  Online  Simulations:  A  Knowledgebase  of  100+  Simidation 
Development  Tools  and  Services,  Sunnyvale,  Calif.:  Brandon  Hall  Research,  2006a. 

- ,  Power  Point  to  E-learning  Development  Tools:  Comparative  Analysis  of  20 

Leading  Systems,  Sunnyvale,  Calif.:  Brandon  Hall  Research,  2006b. 


201 


202  Making  Improvements  to  The  Army  Distributed  Learning  Program 


Clark,  R.  E.,  “Guided  Experiential  Learning:  Training  Design  and  Evaluation,” 
PowerPoint  presentation,  2005.  As  of  October  10,  2006: 
http://projects.ict.usc.edu/itw/gel/ 

Coombs,  J.,  and  M.  Shanley,  “Improving  the  Army’s  Acquisition  Strategy  for 
Acquiring  Interactive  Multimedia  Instruction  (IMI)  Courseware,”  unpublished, 
2008. 

Deparatment  of  the  Army,  “All  Army  Activities”  (ALARACT),  May  6,  2007. 

Department  of  Defense,  Strategic  Plan  for  Transforming  DOD  Training,  February 
2009. 


- ■,  Training  Transformation  Implementation  Plan,  February  2006. 

Distributed  Learning  System,  System  Requirements  Specifications:  Increment  3, 
V2.1-8,  Submitted  by  IBM,  September  8,  2005. 

Dumond,  J.,  et  al.,  Velocity  Management:  The  Business  Paradigm  That  Has 
Transformed  U.S.  Army  Logistics ,  Santa  Monica,  Calif.:  RAND  Corporation, 
MR-1108-A,  2001. 

http://www.rand.org/pubs/monograph_reports/MR1108.html 

Goldstein,  I.  L.,  “Training  in  Organizations,”  in  M.  D.  Dunnette  and  L.  M. 
Hough  (eds.),  Handbook  of  Industrial  and  Organizational  Psychology,  Vol.  2,  2nd 
ed.,  Palo  Alto,  Calif.:  Consulting  Psychologists  Press,  1991. 

Hallmark,  B.  W.,  and  S.  J.  Gayton,  Improving  Soldier  and  Unit  Effectiveness  with 
the  Stryker  Brigade  Warfighters’  Forum,  Santa  Monica,  Calif.:  RAND  Corporation, 
TR-919-A,  forthcoming. 

Hammer,  M.,  and  J.  Champy,  Reengineering  the  Corporation:  A  Manifesto  for 
Business  Revolution,  New  York:  Harper  Business,  1993. 

Headquarters,  Department  of  the  Army  (HQDA),  Materiel  Requirements, 

AR  71-9,  April  1997. 

- ,  Army  Acquisition  Policy,  AR  70-1,  December  2003. 

- ,  Army  Knowledge  Management  and  Information  Technology,  AR  25-1,  July 

2005. 


-,  Counterinsurgency,  FM  3-24,  December  2006. 

-,  Army  Campaign  Plan— Change  5,  April  2007. 

-,  Enlisted  Promotions  and  Reductions,  AR  600-8-19,  July  2007. 

-,  Army  Training  and  Leader  Development,  AR  350-1,  August  2007. 

-,  Operations,  FM  3-0,  February  2008. 

-,  Army  Training  and  Leader  Development  Strategy,  draft,  March  2008. 


Bibliography  203 


- ■,  “TSPU  MDEP:  Training  Support  to  Units,”  briefing,  March  19,  2008. 

- ,  Memorandum,  SUBJECT:  Army  Knowledge  Management  Principles,  July 

2008. 


- ,  Training  for  Full  Spectrum  Operations  (Publications  Draft),  FM  7-0, 

December  2008. 

Headquarters,  Combined  Arms  Center,  OPORD  08-050A,  Formation  of  CAC- 
Knowledge,  February  2008. 

“IBM’s  Learning  Transformation  Story,”  June  2004.  As  of  January  2011: 

http://www-304.ibm.com/jct03001c/services/learning/solutions/pdfs/ 

learning_transformation.pdf 

Kirkpatrick,  D.  L.,  “Techniques  for  Evaluating  Training  Programs,”  Jotirnal  of  the 
American  Society  of  Training  Directors,  Vols.  13-14,  1959-1960. 

- ,  Evaluating  Training  Programs:  The  Four  Levels,  San  Francisco,  Calif.: 

Berrett-Koehler,  1994. 

Mayer,  R.  E.,  “Strategies  Based  on  the  Modality  Principle,”  in  H.  F.  O’Neil,  What 
Works  in  Distance  Learning,  Report  to  the  Office  of  Naval  Research,  Center  for  the 
Study  of  Evaluation,  University  of  California,  Los  Angeles,  February  28,  2003. 

- ,  and  R.  Moreno,  “A  Split-Attention  Effect  in  Multimedia  Learning: 

Evidence  for  Dual  Processing  Systems  in  Working  Memory,”  Journal  of 
Educational  Psychology,  Vol.  90,  1998,  pp.  312-320. 

Northern  Regional  Contracting  Center,  Army  Contracting  Agency,  The 
Distributed  Learning  Education  and  Training  Products  Contract,  contract 
W911S0-06-D-0001,  delivery  order  W911S0-06-D-0003-0008,  December  8, 
2005,  and  associated  contractual  documents,  Fort  Eustis,  Va. 

O’Neil,  H.  F.,  What  Works  in  Distance  Learning,  Report  to  the  Office  of  Naval 
Research,  Center  for  the  Study  of  Evaluation,  University  of  California,  Los 
Angeles,  February  28,  2003. 

Office  of  Federal  Procurement  Policy,  Lire  Federal  Acquisition  Regulation,  48  CFR  1. 
As  of  February  28,  2008: 
http://acquisition.  gov/ far/ 

Piskurich,  G.,  P.  Beckschi,  et  al.,  The  AST D  Handbook  of  Training  Design  and 
Delivery,  New  York:  McGraw-Hill,  2000. 

Rossett,  A.,  E-Learning  Handbook,  American  Society  for  Training  and 
Development,  New  York:  McGraw-Hill,  2002. 

Schank,  R.  C.,  Designing  World  Class  E-Learning:  How  IBM,  GE,  Harvard  Business 
School  and  Columbia  University  Are  Succeeding  at  e-Learnim,  New  York:  McGraw- 
Hill,  2002. 


204  Making  Improvements  to  The  Army  Distributed  Learning  Program 


Schirmer,  P.  C.,  J.  C.  Crowley,  N.  Blacker,  R.  Brennan,  H.  A.  Leonard,  J.  M. 
Polich,  J.  M.  Sollinger,  and  D.  M.  Varda,  Leader  Development  in  Army  Units:  Views 
from  the  Field,  Santa  Monica,  Calif.:  RAND  Corporation,  MG-648-A,  2008. 
http://www.rand.org/pubs/monographs/MG648.html 

Secretary  of  the  Army/CSA  Memorandum,  “Subject:  Army  Knowledge 
Management  (AKM)  Guidance  Memorandum  Number  5 — Army  Training 
Enterprise  Integration  (ATEI),”  September  7,  2004. 

Senge,  P.  M.,  The  Fifth  Discipline:  The  Art  and  Practice  of  the  Learning 
Organization,  New  York:  Doubleday  Business,  1994. 

Shanley,  M.  G.,  J.  C.  Crowley,  M.  W.  Lewis,  R.  Masi,  S.  G.  Straus,  K.  J. 
Leuschner,  S.  Hartman,  and  S.  Stockly,  Transformation  and  the  Army  School 
System,  Santa  Monica,  Calif.:  RAND  Corporation,  MG-328-A,  2005. 
http://www.rand.org/pubs/monographs/MG328.html 

Shanley,  M.  G.,  J.  C.  Crowley,  M.  W.  Lewis,  R.  Masi,  K.  Leuschner,  S.  G. 

Straus,  andj.  P.  Angers,  Supporting  Training  Strategies  for  Brigade  Combat  Teams 
Using  Future  Combat  Systems  (FCS)  Technologies,  Santa  Monica,  Calif.:  RAND 
Corporation,  MG-538-A,  2007. 
http://www.rand.org/pubs/monographs/MG538.html 

Straus,  S.  G.,  M.  G.  Shanley,  R.  M.  Burns,  A.  Waite,  and].  C.  Crowley,  Improving 
the  Army’s  Assessment  of  Interactive  Multimedia  Instruction  Courseware,  Santa 
Monica,  Calif.:  RAND  Corporation,  MG-865-A,  2009. 
http://www.rand.org/pubs/monographs/MG865.html 

StrykerNet  Staff,  “SBCT  Warfighters’  Forum  (SWfF)  Information  Brief,” 
presentation,  May  2010. 

TRADOC  Inspector  General  Report,  Soldier  Training  Review,  December  2005. 

U.S.  Army  Combined  Arms  Center,  FMI  6-0-1,  Knowledge  Management  Cell, 
October  2007. 

U.S.  Army  Combined  Arms  Training  Center,  AC-Knowledge,  February  2008. 

- ,  Organization  and  Functions,  TRADOC  Regulation  10-5-4,  February 

2006. 

U.S.  Army  Training  and  Doctrine  Command  (TRADOC),  Organizations  and 
Functions,  TRADOC  Regulation  10-5-1,  Final  Approved  Draft  Version,  undated. 

- ■,  The  Systems  Approach  to  Training  Management,  Processes,  and  Products, 

Fort  Monroe,  Va.:  TRADOC  Regulation  350-70,  March  9,  1999.  As  of  January 

2011: 

http://www.tradoc.army.mil/tpubs/regndx.htm 
- ,  TADLP  Campaign  Plan,  June  2001. 


Bibliography  205 


- ,  Operational  Requirements  Document  (ORD)  for  a  Modernized  Training 

System,  Fort  Monroe,  Va.:  TRADOC,  revised  October  22,  2002. 

- ,  Multimedia  Courseware  Development  Guide,  Fort  Monroe,  Va.: 

TRADOC  Pamphlet  350-70-2,  June  26,  2003.  As  of  March  2011: 
http://www.tradoc.army.mil/tpubs/pams/p350-70-2.htm 

- ,  Distributed  Learning — Managing  Courseware  Production  and 

Implementation ,  Fort  Monroe,  Va.:  TRADOC  Pamphlet  350-70-12,  March  29, 
2004.  As  of  March  2011: 

http://www.tradoc.army.mil/tpubs/pams/p350-70-12.htm 

- ,  Army  Comprehensive  Guide  to  Modtdarity,  October  2004. 

- ■,  TRADOC  Campaign  Plan  with  Change  1,  June  2006. 

Warded,  C.,  “Levels  of  Interactivity,”  paper  presented  at  2006  dL  Workshop, 
March  14,  2006. 

Wexler,  S.,  et  al.,  E-Learning 2.0. ,  Santa  Rosa,  Calif.:  The  ELearning  Guild,  2008. 

Woodhill,  G.,  Emerging  E-Learning  Technologies:  Tools  for  Developing  Innovative 
Online  Training,  Sunnyvale,  Calif.:  Brandon  Hall  Research,  2007. 


